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Abstract 

4 

This, report describes the evaluation plan for the 1973-74 
scale field test of the Comprehension element of the Wiscong ' 
for Reading Skill Development arid prese^^ results for the 
test objectives investigated. 

The field test was conducted in ten elementary CK-S) scr 
in Wisconsin and one in Cololrado. Eight of the schools were 
small-city and suburban areas 'where reading achievement is ty^iCTlIy at 
or above grade level (site type A), while the remaininf two were urban 
schools where reading achievement is typically below grade level ^site 
type B)^ Site differences also included school organization, whether 
multiunit or conventional, and the number of Design elements being 
implemented at that time. • ' ^ 

Field test objectivfes -1 to 3 pertained to prodti^l usability; results, 
on the whole, were positive: ^ 

1. Product implementation — attainment of the ten developer-stated 
requisites for effective iiDt>lementation Vas' generally satisfactory in 
th6 site typ^ A schools for all requisite and in the site type B schools 
^r 'eight of the requisites. ;. ^ \ > , - . 

2. ^ Product feasibility—school staff^ of both site types considered 
the product a viable program for eleme'ntcid^ schools. 

3. Product deficiencies — school staff members .identified errors, 
omissions, and confusions in the product's materials. 



V. 




Objectives 4 to' B«rwere concerned with product: effectiveness. 
Measuremeiit' employed both program-embedded and standardized tests. The 
evaluation design for objectives 4 and 5 utilized a 'cross-sebtional 
comparison of the performance ^of learners of the same age/grade level in 
the field t^t sites in different academic years — immediately prior to 
product installation (Fall 1973, preimplement'ation group) and near the 
end of the first year of implementation (Fall 1974, implementation 
group). Each age/grade le^vel pre implementation group served .as the : ; 
control for the group of learners at the same age/grade level one yeaf 
later. Longitudinal comparisons were also made with the standardized; 
measures. . 

Results for objectives 4 to 8 are reported according to site type: 

■ ._,» • 

4. Cross^sectional comparative performauice--program-embedded 
tests — ccanparisons were positive for 22 of the 28 Design objectives 
assessed in the site, type A schools and for 21 of the 29 dbject^^ves 
assessed in the site type 3 ^hools. 
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5. cross-sectional comparative performance—standardized reading 
achievement tests~«in»-i«-*aw score means occurred at all grade levels 
tested in the site type A sch^ls and at four of five grade levels , 
tested in the site B schools. • ' ■ . . , , ^ ^ \ 

6 Longitudinal comparative performance of individual students- 
standardized reading achievement tests— across three grade level com- 
parisons for the site type A schools, 15.4% of the students experienced 
a loss in grade equivalent, 5.4% exp-ferienced no change in grade equi- 
valent scores, 27.9% a gain from a month to a yeaj^and. 51^3% a g^n^ot^ 
1.^1 or better. .Across two grade level cooiparisons^-ttjr the site type 
, schools, 23.2% of the students experienced a loss in gVa 
'7.2% experienced no -change im glsade equivalent scores, 
a monSTt^-^^W, and 36.2%Na gkiti of 1.1 or better. 

7.Nperceht masters afU^r skill instructicsn— the irand total per- 
cent masted across all of the Design Comprehensioy ski^ was 71.9% for 
the site typte A Sfchools and 60.3% for the site tyjpe B sch^x^^. 

8 Skill retentipn— the overall retention bate for skKUs mastered 
at break-in testing or after instruction was 85.^% for the siXe type- A 
schools and 77. "2% for the site type B schools. 



ade jequiva^ent , 
33.3% a gain from 





i . 



THE PRODUCT 



OVERVIEW 

The Wisconsin Design for Reading Skill Development (Design) is an 
oJjjective-lM^ped system that provides both structure and substance for an 
elementary school xeading program, kindergarten .through grade six. The 
Design provides the basis for a skill-oriented approach to the teaching 
of read ing and is based on the assumption that if learners master essen- 
tial subskills, they will be successful readers. The four majorfiesign 
components are (1) skills and objectives, (2) assessment mater|Sls; (3) 
instructional resources; and (4) management techniques and mat^^rials. 



Skills and Objectives 




The structure of the Design is provided by its Outline of Reading 
Skills (Otto 61 Askov, 1974) , a scope and sequence description of what 
the developers believe to be essential reading skills for the elanentary 
school. The skills are grouped into six main areas: Word Attack, Com- 
prehension, Study Skills, self-Directed Reading, Interpretive Reading, 
and creative Reading. Each area is subdivided into levels that corre- 
spond roughly to grade levels as shown in Table 1. 

An instructional objective has been developed for each, skill--45 
objectives for Word Attack, 36 for Comprehension, 71 for Study Skills, 
15 for Self-Directed Reading, 18 for Interpretive Reading, and 14 for 
creative Reading. Two types of instructional objectives have been 
developed for the six Design skill areas. There are behavioral, or. 
prescriptive, object,ives for the Word Attack, Comprehension, and^tudy 
Skills areas ; and th^re are expressive, or descriptive, objectives for 
the self-Directed, Interpretive, and Creative Reading areas. A prescrip- 
tive objective specifies what a child is to do^ to attain mastery of the 
skill-' it is written at a mid-level of specificity and assumes an 80% 
criterion for mastery. The expressive objectives are quite different in 
that they are mainly descriptions of activities that are judged to be 
relevant to the development of skills in the Self -Directed, Interpre- 
tive, and creative Reading areas. Expressive objectives permit focus on 
the important aspects of reading instruction for which there are intended, 
vet unassessaUale, results. , « 

A terminal objective has been stated for the Word Attack, Compre- 
hension, and Study Skills elements (see the respective ^^eacher s planning 



; TABLE 1 

DESIGN SKILLS BY AREA AND BY TRADITIONAL GRADE LEVEL 



Skill Area 


K 


1 


2 


3 




4 


5 




6 


Word Attack 


A 


B 


c / 


D 




- 


-/ 






Comprehension 


A 


B 


C 


D 




E 


F 




G 


Stiady Skills 


A 


B 


C 


D 




' V 


F 




G 


Self-Directed Reading 




A-C 






D-E 






F-G 




Interpretive Reading 




A-C 






D-E 






F-G 




creative Reading 




A-C 






D-E 






F-G 





^guides for the terminal objectives of Word yttack and Study Skills) . 
The terminal objective states the student pbtcome expectations once all 
of the instructional objectives have been mastered. 

Assessment Materials ^ ^ 

Criterion-rreferenced assessment materials have been developed to 
assess the behavioral objectives in the Word Attack, Comprehension, and 
Study Skills areas. There is a. test to assess each objective. The 
Wisconsin Tests of Reading Skill Development (WTRSD) and performance 
tests comprise the formal assessment componjfent. The WTRSD are paper- 
and-pericil tests which can be group administered and scored either by 
hand or by computer. They are available in two parallel ' forms and in 
either hand-scorable> single-sheet format (for a single objective) 
or machine-scorable, booklet format (for all the objectives at a given 
level)-. Typically, the booklets are used for break-in testing to find 
the students' initial instructional levels, and the sipgle-sheet tests 
are used after skill iriBkuction'to assess individual attainment of 
criterion performance. Tferformance tests, developed for the behavioral 
objectives that cannot be assessed by writttfh means, are individually 
administered' and require oral and/or motor responses. Manuals for test 
administrators are a part of the assessment materials. 

informal assessment exercises, called Guides ^to Informal Individual 
Skill CJbseirvation (Otto s Askov, 1974), are also available for the Word 
Attack skills. The guides provide directions for observing learner 
behavior. They are criterion-referenced in the same sense that the 
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3 



WTRSD and the performance tests are; they are designed to sample criteric^ 
behaviors that are prescribed by objectivep. In practice, these guides - 
serve as supplem^ts to the formal teats.* Where the formal tests bring 
objectivity and reliability to the assessment process, the informal 
guides offer flexibility and, in some iiistances, a different way to 
observe a skill. ^ S 

In the Self-Directed, Interpretive, and Cre#itive Reading areas 
there cu:^ no criterion-referenced a^essment materials. Instead, 0 
teachers are urged to plan systema'tlc exposure of each leaamer to all of 
the skills and relatept- /activities over relatively long time periods emd , 
to observe and judge perfoxnnnance in view of the general descriptive 
statement. Since mastery,, per se, is not the goal, the evaluative task 
involves informal observations/ upon \i^at has been produced as the learnei^^ 
is exposed to a skill. ;/ ^ 

InstAictional Resources j 

Bdbause a vast arr^y of commercially-published m^erials is 
already available for teaching most of the essential skills, the Desigri 
includes a component .called the teacher's resource file, which is a means 
for organizing existing materials amd activities. There is a teacher's 
resource file. for f4ch skill area which has behavioral objectives — Word . 
Attack ,'^CcMi5>rehen9lon, and Study Skills. Within each file, materials 
fiuid activities euf'e identified and organized by objective. On the inside* 
of each file folder are printed the behavioral <A)jective for a peurticular 
skill and a list of coimnercially-published printed, audiovisual, and 
multimedia materials that are useful in teaching the skill. The commercial 
materials listed represent only a limited sampling of the vast array of 
materials availsdDle. Teacher-directed activities are provided as inserts 
to each folder and are intended as guides in planning skill instruction. 
Thus, in each file there is a collection of materials emd activities 
appropriate for teaching the criterion behavior (s) prescribed by each 
objective. The collection is viewed as a "stairter set" to which teachers 
are expected to add whatever local resources they judge to be relevant 
to the objectives and to the needs of their pupils. 

There is a single teacher's rdsoiurce file for the Self -Directed, 
Interpretive, and Creative Reading arecis. Its organization is similar 
to that of the files for the other areas, with a specific folder for 
each skill strand at each level. The main difference is Jthat the material 
related to each skill is organized to provide breadth of exposxure rather 
than mastery of prescribed criterion behavior. The teacher's task in 
Self -Directed, Interpretive, and Creative Reading* is to give learners 
opportunities to develop the skills through situations in which they 
brir^g to bear personal meanings and express their individuality. 

Mamagement Techniques amd Materials 

Three components of the Design are directed specifically at pro- 
viding assistance with its management and implementation. The Rationale 
and Guidelines (Otto & Askpv, 1974), which describes all six skill areas 
of the Design , provides a rationale for the development of the Design 
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amd guidelines for its iropleijientation. There are separate teacher's 
planning guides for Word Attack,, for comprehension, and for Study Skills, 
ahd one guide fbr Self-Directed/ Interpretive, and Creative Reiading. 
The planning guides include' the specific information teachers need to 
implement an objectiverbased approach to each skill area. Taken to- 
gether, the Rationale, and^ Guidelines and the teacher's planning guides 
provide the practical assistance that will enable a school staff to plan 
and carry out the assesatient, grouping, sched^ling, and record keeping 
that is required for* successful inqplementation. 

"A final management component is the profile card/ Prof He* cards 
are provided in Word Attack, Comprehension, <*nd Study Skills for keeping, 
a current record of ea||t individual learner's skill development status. 
On. the Comprehension prclfile card, for example, all of the Design Ccmpre- 
hension skills are listed by level; a corresponding hole by each skill/ 
i& notched open when the learner achieves mastery of that skill. When • 
current, the profile card shows which behavibral objectives a learner 
has and has not attained at a given time. The- one ^record, card for Self-» . 
Directed, Interpretive, ahd Creative Reading is designed for recording 
the number of exposures a • learner has had to each of the descriptive 
objectives. 

In suOTiary, the Qesign , with its four major components—sHills and 
objectives, assessment materials, instructional resources, and manage- 
ment techniques and mdt;erials— -is a model for focused reading instruc- 
tion.* The Design is co^eptualized to identify essential skills, specify 
these skills with objectives, assess ^^W|»ers* skill deve;.opment, organize 
instructional materials and prdcedxir^ffljnage instruction, and monitor 
learners' progress. The Rationale and^Wldelines (otto & Askov, 
should be consulted for a complete description of the program. 

The Design has been under development since 1967; during this 
period it has undergone revisions based upon information gathered from 
content and measurement experts, teachers, evaliiators, and several ftield 
tryouts. A large-scale field tryout of the Wprd Attack element was 
conducted in 1970-72, and of the Study Skills element in 1971-73. A 
pilot test of the Self-Directed, Interpretive, and Creative Reading 
elements occurred in 1972-73. All- elements except Comprehension are now 
available in commercial edition. 

The Design provides a framework for an effective skill development 
program in reading, and 4:his framework is compatible with the program of 
the Wisconsin Rese«irch and Development Center for Cognitive Learning, 
Individually Guided Education (IGE) . IGE is a comprehensive alternative 
form of schooling. Built from research findings and actual tryouts in 
school settings, IGE provides an educational experience for each child 
which takes into accovint what he already knows, how rapidly he learns, 
and how he goes about learning. To help schools achieve this goal, IGE 
provides for changes in the total educational system. 

The Instructional Programing Model (IPM) , perhaps the most crucial 
element of IGE, helps schools focus on individual needs. The IPM pro- 
vides a framework for setting educational objectives; selecting a range 
of specific instructional objectives; assessing learners' performance; 
letting objectives for each learner; planning and implementing instruc- 
tion via activities, materials and media, and time, space, and equip- 
ment; and reassessing learners* attainment of objectives. The prin- 
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ciples of the IPM* are emboOTe"^ in the Design . Implementation of IGE 
through use of the Design involves assessing each learner's skill develop- 
ment^ grouping learners who need .*o develop the same skill or conf ig- / 
juration of skills, ^ providing^ individual assistance as necessary within'- 
each group to ^ake into account differences in rate and style of learrtihg, 
providing for^ independent activity or study, reassessing, and regrouping 5 
as some learners \evelop the configuration of skills, or paurt of them, • 
and others do .not A The four major .components of the Design , previoijysly 
described, facilitate ij^plementation of the Instructional Progrtming 
Model element of IGE^ * • 

OBJECTIVES F0R THE COMPREHENSION ELEMENT ' ' ^ 

Thirty-six Con?>rehen9'ion skills wefte identified after a' careful 
survey of tiwj, prof essional literature, assessment 'amd. instructional 
materia}.s; an^ standardized tests in the area of (jomprelfiensipn. ywo 
basic assumptions distinguish the Design 's rtodel of Comprehension from 
other models. -First; the developers found it nece8S2ury to distinguish 
between skills requiring converfcfent thinking and those requiring di\ 
vergent thinking. The Design Con^rehens ion model deals only with those 
skills which require convergent thinking, • i.e. , there are right, and 
wrong amswers. Skills that encourage divergent thiWcing are appropri- 
ately placed in the Design 's Interpretive and Creative Reading areas 
where the descriptive objectives allow more latitude in responses. 
Divergent thinking offers no right or. wrong answers; it involves 
questions which have answers that are open to interpretation. 

The second distinction between the Design model aryi that offered in 
traditional materials occurs in the treatment of inferential thinking. 
Many programs offer a skill commonly referred to as "Inferential Thinking." 
The developers, however, found that inferential thinking cuts across all 
skills and levels.* In the Design , therefore, inferential skills are not , 
treated independently but are embodied in separate skills, e.g., inferring- 
the main idea, inferring an outcpme, etc. 

■* • 

Specific Objectives • 

The Comprehension objectives are organized by strand and by level. 
Each strand includes related skills which recur at higher levels with 
more complex behavior expected at each successive level. The skill 
levels correspond generally to the grade levels shown in Table 1. Ther^ 
are six strands and seven levels. The matrix of skills for each Strang 
and level is shown in Table 2. Some of the strands have more than on^ 
skill at, a particular level. For example, there are two reasoning 
skills at each of Levels A, F, and G. Also, there is not always a skill 
for each strand at each level. For example , there are no Detail skl/lls 
at Levels A and G; Context skills occur only at Levels D, E, and GJ The 
levels are arranged in sequence. Within a level, however, there is no 
hierarchy of skills. The skill number designations shown in Table 2 are 



17 



TABLE 2 , 

MATRIX dF COMPREHENSION SKILL NUMBERS 
' BY STRAHD AND LEVEL ' 





^ - . ■ • 










Level 










A - 






. u 
























Main Idea ' 


1 . 




1 


1 


1- 


r 




S^cjueiice 


2 


2 


2 


2 


2 ~ 


} 


1 


Reasoning 


' 3 
4 


3 


3 


3 . 


"3 ■ 


3 
4 


,.3 ' . 


Detail 




r4 


,4 




4 


5 ' 










5 


5 




s)' 


,K • 


Context 








'6 . 






■ 4 ■ * 


Affixes . 










, 6 


6 


5 



for reference piarposes only; they are a convenience for school personnel 
since teachers usually, deal with skills within one level at a time. 
Information' in some of the later tables^ and t^xfc refers to the skills by 
level and number rather tham by strand and skill name. 

The "Statement of Skills and Objectives for Comprehensipn," small- 
scale field test edition, is presented in Appendix A. Chapter 2 of . the 
Comprehension teacher 's planning guide '(Chester^ Askov^ Hudson, &^tto, 
1974) gives a con5)lete description of the skills and their organization 
for the large-scale^ field test edition^ which .is not subsl^tially . 
different from the/small-i^cale field te:st edition. 




COMPONENTS ^„ 

The following materials, ^ich are described in the Overview, 
available for use dxiring the field test: 

Wisconsin Tests of Reading Skill Development :, Comprehension 
Machine-scprable booklet edition, l^evels A-D, Form I 
Hand-scorable separate .edition^ Levels A-D, Forms I and II 
Reusable booklet edition. Levels E-G, Pormfi I and II 
Machine-scorable answer sheets. Levels E-G, Forms I and II 



Hand-scorabla answer sheets, Cevels E-G, Forms I and II 
test Administrator ' a Manuals, Levels A-(3 
Teacher's' Resource Fjle ; Comprehension , Levels A-G 
Teacher's Planning Gui,de ; Comprehension 
Profile cards: Compreft^sion . \ . 

\ * * . ■ / . ■ 

REQUISHES FOR EFFECTIVE IMPLEM^^^'^"'''" ^ ' ^ 




Certain? conditions were consfoeired by the developers ,to X}e neces- 
sary for effective inrplementation.. ^en requirements for installation - 
and implementation were stated for(tAf field t^t' participants. Requi- 
sites 1 to 3 pertain to product i/istaXlation, \(*iile requisites 4 to 10 
pertain^to the instructional, assess^nent, and 6ianagement aspects of 
inq^lemeAtation . • • '/ 

; 1. Locdl representatives of the field test schogls must attend a 
one-day Center-sponsored inse^vice- to' ac<tuir^ information regarding 
implementation of ^;Wie product . . 

2. Within one month following the Center iftservice, local school 
representatives must conduct inservice sessions for local faculties to 

^ provide iriformatioxi and support for implementing the product. The 
•« agenda for tj^e sessions shoul^vinclude- a description of tihe^rodUct, 
strategies for scheduling instruction, and opportunities for.. keying 
loccil .materials to the product's behavioral objectives. 

3. Initial break-in testing must be ^conducted to identify initial 
injfcructional levels for learners i^rades one to six (including kinder^ 
gaften second semester) . Retesting^u^ be conducted within three weeks 

^ at ^ higher or lower level for learners who were tested at an inappro- 
priki4 break-in level. New learners must receive break-in testing 
within one month after entering school. 

4. The local faculty must dienpnstrate a commitjnent to the xJompre- 
hension element that ens\ires that '^vea*y learner receives instruction 
appropriate ♦ to his skill deficiencies. j?^vwv«?s^' 

5. Instructional skill groups must be formed on- the bei^i^^'oi 
assessment information. 

6. Instruction must be provided that is related to the skill de^ 
ficiencies of the instructional groups. Two hovirs per week of skilly 
instruction is the minimum time requirement. ^ ^ 

7. ' Learners must be postassessed following instruction in order to 
ascertain their mastery level. ^ 

8. Learners ^must be regrouped in a manner that is consistent with 
their current skill needs; . 

9. Profile cards must be kept up-to-date ai(d in a fashion that 
ensures their' utility in instructional decisions. 

10. New skill groups must be foi*med at a rate that ensures com- 
pletion of at least one level .per grad^* in a school year throughout the 
field te^t period. 
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PRIOR PRODUCT EVALUATION • . ^ 



A pilcvfe test of the product was conducted 5brA[>g the spring semes- 
tei: of the 1972-73 school yeeu: for Levels A-C arwi /Airing the fall semes-' 
ter of the 1973-74 school year for Levels D-G. /*%iree Wisconsin schools 
were identified as pilot test pcu^ticip^nts. Two of the schools, one 
located in' a rural 'community and the other in a small bi'ty, were multi- 
unit in orga^iization. The third, an inner-city school, was conventionally 
orgfimized. All three were eaQ)erienced Design users, each having implemented 
the Word Attack element for a minimum of two years'? two of the schools 
had implemented the Study Skills element as well. ' 

Both prior to and during the pilot test,' the Comprehension assessr 
n^i^ component underwent several stages of evaluation. Since the 
iffetruments are criterion-referenced^ test specifications followed' from 
the Comprehension behavioral objectives wit^ which the tests were to be:| 
u^d. . The behavioral objectives theinselvM^^^ been reviewed by expert* 

the fields of reaiSing and measureinent.^^^ 9»<||^P administer abl|||(||,,^ 



in 



fopas were developed for each t^|^ 
appropriately assessu^ by an infil^^i^jp 
* that did not involvesJWritten res^hs< 



tests were developed for 35 of the 36^^|^§^^^^ parallel 
fopns were developed for each t^^^;^jgi^^ more^ 

Irdmll^OTered performance test 

content and ' measurement eatperts Itii^Ji^J^ of objectives, 

appropriateness of item format,^ cllbr^^' Of item intent, .appropriateness 
for audience, clarit^rpfij test directions, appropriateness of assessment 
time, and acceptability of test sta^%ics. 

Test development ,pj?pceeded( irtvafedordan with the evaluation questions 
just specified and consisted of. a^jleaist six steps. Each step included 
a developmental phase and a review phase. The steps were (1) develop- ^ 
ment.of test specifications and sample items,, (2) writing of sufficient 
itei!iis.^to yield bne testii^torm, (3) pilot administration of those items to 
' a , smallj^roup of learners, (4) formal administration^^^ of revised items to 
a number^of learners large enougl^t?6 gather item data, (5) assembly of a 
test administrator's manual, emd (6) use of the instrument under normal 
conditions by a number of^school^jfor classes. Step ^3 was conducted, not 
in the pilot schools, but in a scaool close to Madisorv and the Center. 
•Step 4 was conducted in i:he three pilot schools and i^^a number of other 
schools as well because a large number of students and ""S great deal of 
time were required for the tryouts. Step 6 was a pilot school activity. 

.•The pilot test included fi'el^ test activities in addition to the 
assessment tryouts just described. Two Center-sponsored inservices (one 
fbr Levels A-C. and the othe^r for Levels D-G) provided cin initial tryout 
of ihservice inaterials developed specifipally for the Comprehension 
element. Revisions in the inservice materials were made on the basis of 
teacher feedback. The assessment and msinagement materials and instruc- 
tional resources needed for implementation were used in the^ tryout 
edition. Periodic monitoring visits to the pilot schools were made by 
Center staff members to examine problems involving product implementation 
.and to answer questions regardijig materials and procedtires. , Formative 
data gathered thro^h staff interviews, questionnaires, and pilot school 
meetings identified problem areas. 
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In summary, ptior product evaluation activities included expert 
review of the skill s^€md objectives, evaluation-embedded development and 
tryout of the criterion-referenced tests, and a pilot test during which 
formative data were collected regarding usability of the component 
materials; ' , 

i • . . . • 



0 



FIELD TEST PLAN 



OBJECTIVES' 




,The 1973-74 small-scale f ield test was designed^ to>-^ather product 
usability information and product effectiveness infonnat|>.6n from learner 
behaviors. The data obtained were utilized in making rfeyisiolis for the 
'developmental edition for the 1974-76 large-scale field; jtest. The data 
also identified areas needing further product: refinement; in preparation 
for Uie cOTimercial edition. The field test objecti^esrj grouped under \ 
'^product usability arid pi^oduct effectiveness, follow: / ; 

Product Usability * ' 

1. To learn whether the teacher's resource files. Teacher ' s 
Planning Guide , and Center-sponsored inservice provided sufficient 
information for field test faculties to implanent the product according 
to the requisites for effective in5>lementation. (Product Implementation) 

2. To learn whether the field test faculties considered the product^ 
a viable program for elementairy schools. (Product feasibility) 

t ■ * ■', . ■ • _ 

3. To identify specific errors, omissions, and confusions in the 
product's components, (Product deficiencies) 



Prodifct Effectiveness / ^ ^ 

4. To compare on the product's objectives the mastery levels of 
the in?>lementation group versus the preimplementation group. (Comparative 
performance— r>program-embedded tests ) % ' , 

% 5, To compare on stemdardized tests related to Comprehension the 
scores of the implementation grot|p versiis the preimplementation group,. 
(Comparative Performance — standardized reading achievement tests) 

6. To determine whether implementatiqn groups increased their 
achievement level on standardized tests related to Comprehension more 
than would be expected (more than 1.0 year|i) . (Comparative performance 
olM individual learners — standardized readlpg achievement tests) 
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7. To ascertain the percentage of learners mastering the product's 
objectives at an 80% level immediately after instruction in those skills. 
(Percent masiters after skill instruction) 

\^ 8. To ascertain learner retention rates on the product's objectives 
with an eight-week intermission between skill mastery and retesting. 
(Skill retention) 

Evaluation of field test objective 1 provides usability information 
in relation to implementation, which in. turn has implications for the 
evaluation of product effectiveness.' The usability information pertains 
to the sufficiency of the component materials to yield practices that 
are in accord with the developer's requisites for ef feet ivie implementation 
The data then obti^ined with respect to implementation practices provide ; 
the context for evaluation of the effectiveness data. Product effective-, 
ness, there fore r must be viewed in^jrelation to the degree of product • 
inqplementation. — ^ . ^ 

Objectives- 4, 7, and 8i pertain to effectiveness of the product's 
specific objectives. While Objectiye 4is a direct effectiveness 
measure # 7 and 8 were investigated to acquire information about skill 
mastery and retention rates, information not collected during prior 
Design , field tests, but of interest to both consumers and the professional 
audience. Attainment of the product's terminal objective was left un- 
examined because it was not expected that learners would attafti mastery 
of all the product's objectives by the end of the small-scale field 
test. 

Cost information was not collected because the Center, rather than 
a^ommercial vendor, produced and distributed the product's component 
materials for the small-scale field test. 



PARTICIPANTS 



ilementary (K-6) schools, nine in Wisconsin and one in Colorado, 
^^rt>£ipated in the small-scale field test. An attempt was made to 
include schools serving ^t>opulations with varying achievement and socio- 
economic characteristics, including inner-city urban schools and sub- 
urban and small-city schools. A, second variable of interest was the 
school's organization, whether multiunit or conventionally organized 
into self-contained classrooms. A third variable of interest was , the 
number of Design .elements being implemented by the school. Distribution 
of the field test sites with reference to these variables is simanarized 
in Table 3. Although Table 3 shows a total of 11 schools, only 10 were 
formal field test participants. The additional school (site type A, 
which had a multiunit organization, and was implementing W6rd Attack and 
Study Skills in addition to Con^rehension) was included in the evaluation- 
related testing program (objectives 4 to 6) , but its implementation iDf 
the product wAs not monitored. Sufficient information was known about 
its implementation practices, however, to include its data in the analyses 
for objectives 4 to 6. 
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TABLE 3 



DISTRIBUTION OF FIELD TEST SITES' 















Reading 
Achievement 


• , School Orgcmization 
MULTIUNIT CONVENTIONAL 
Design Elements^ Design Elements > 

b ' b . b 
WA WA, SS WA WA> SS 


Total 


Site Type A: 
Typically at or ^< 
above grade ^evel 
(subiirban, small-city) 


3 


5. 




9 - 


Site Type B: 
Typically below 
grade level (\irban) 






1 1 


2 


Total 


3 


5 ; 


1 . 2 


11 


a 

In addition to Comprehensipn 








b 

= Word Attack; SS = 


Study Skills 







As shown in TcO^le 3, two schools were in an urban area where 
reading achievement is typically below grade levels aind nine were in 
small-city and suburban areas where reading achievement is typically at 
or aibove grade level. ^ Tworthirds of the schools were multiunit in 
organization; Inclusion of an urban multiunit sciiool was not possible at 
the time of the field test. All schools had been Ixnplementing the 
Design 's Word Attack element for two to three years prior to beginning 
Comprehension Implementation and more than half were also, implementing 
the Study Skills element. 

The following method was used to identify the sites. A prereq- 
uisite requirement for participation in the field test was prior imple- 
mentation of th^ Desi^Qi Word Attack element. The developer's recom- 
mendation for installation and sequence is that Implementation generally 
begins with assessment and teaching of the Word Attack skills since they 
are basic to all other reading skills. 

The Design 's Word Attack and Study Skills small-scale field tests 
had created a pool of 22 schools from which to choose, and all of the 
schools in this pool were asked if they wished to participate in the 
Comprehension field test. Because of budgeting considerations, the 
districts with more than one school in the previous Design field tests 
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were informe'd that only one or two schools f^^pm the district could - 
participate. Schools were then selected ^or the categories shown in ^ 
Table 3. A factor in the selection process was a. school's quality of 
commitment during the previous 'i^ield tests. Two of the final ten , 
schools ^elected had not participated in the prior fiield tests. The two 
were xn(ultiunit sl,te type A schools, and they were implementing Word ^ 
Attac)Os£ut jiot; Study Slcills. Two other schools in the final seleot^ion, - 
one site type A and mult^unit, the other a site type B and conventionally 
organized, were asked to discontinue Study Skills implementation during 
the period of the Comprehension small-scale field test.- Schools from 
the pool not selected were permitted to purchase the component materials' 
frcxn the Center at- a cost of $.60 per student, attend the Center-sponsored 
inservice, cuid implement the product*- independently of the field test. A . 
list t>f the field tePSt schools and their characteristics is presented in 
Appendix B. . ' 

The field test schools signed a formal agreement with the Center 
(Appendix C) in which they expressed a commitment to meet the field test 
requirements and in which the Center enumerated the contributed resources. 



ACTIVITIES 

The major field 'test activities, as illustrated in Table 4, included 
inservice, assessment, skill instruction, and monitoring of produQj: 
installation and implementation. The events and schedule were commu- 
nicated to the sites at the outset of the field test. The development 
and.tryouts of materials had not been completed for all of Levels A to G 
by the start of the field test. Therefore, the break- in and baseline 
testing activities and the start of skill instruction were scheduled by 
skill level clusters. The criterion-referenced tests and teacher's 
resource files needed for l^ese activities were available for Levels A 
to D in September 1973, for Levels E cUid F in November 1973,— aftd-^for 
Level Q in JcUiuary 1974. The product' s- other component materials, 
teacher's plcUining guide and the pupil profile card, were availcible 
the schools at the beginning of the field test. 

Activities for monitoring product installation and implementation 
and for baseline and comparison data collection will be described in the 
instnmientation and method section for field test' objectives 1 to 3 and 
4 to 6 respectively. 



Inservice 

A Center-sponsored one-day inservice meeting was held in September 
1973 in Madison to acquaint the field test schools with the product. 
Central office personnel from the paf tibipating districts were invited 
to attend, and each field test school was permitted to send three staff 
members to the training session with the expectation that the attendees 
would then conduct their own local Comprehension inservice. The Center 
inservice agenda'^in^luded sessions on the Comprehension strcuids, sche- 
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TABLE 4 



AGENDA OF FIELD TES^ EVENTS 

V. ■' ' 



Dates 



Field Test Events 



September 1973 
.October 1973 • 



Sept^er 1973 
November 1973 

September 1973 
November 1973 



September 1974 
November 1974 



October 1973 
Jan\iary 1974 
September 1974 



October, November 
.1973; January, March, 
May, October 1974 



Inservice: ' . 

Center-sponsored 
Lodal 

Assessment: 

Break-in listing to determine learners" 9 

instructional, levels 

Levels A-D ^ 
.Le^eXs E-F ' ; . , 
Baseline data, collection on preimplementation 

group for grade levj(ls 

1-3 site type A? 1-4| site type B 

4-$ site type K% 5-5 site type B 
Comparison data collection on implementation 

^roup for grade levels 

1-3 ^ite type A? 1-4 site type B ^ 
4-6 site type A; 5-6 site' type B 

Skill instruction: ^^ ■ 

Commences for Jjcvels A-D ^ ' 

Commehces for Levels E-G 
Coi^tinues for Levels A-G 

Monitoring of product installation and implementation: 
Total of f pur visits to Teach site 



duling for skill instruction, and field test expectations. The complete 
agenda appears in Appendix D. Attendees were provided a packet of 
materials that could be used in their local inservice. The packet 
included descriptive information about- each of the six Comprehension 
skill strands, the field test's objectivesf and schedule of activities, 
and the requisites for effective implementation. 



- Assessment: Break-in Testing 

Break-in tests were administered to students in the field test 
schools to identify learners' initial skill instructional levels and to 
diagnose their skill proficiencies and deficiencies. This placement 
testing is a prerequisite activity for the formation of instructional 
skill groups. Test administration of the booklet editions for break-in 
testing was managed by the field test faculties, and scoring services 
were provided by the Center. Tables 5 and 6 present the break-in 
testing summaries for the schools according to their typical reading 
achievement, whether at or above grade level (site type A) , or below 
grade level (site type B) . Retesting up or down a skill level is 
required when learners master either all or none of the skills at the 



levix first tested, the test level information presented in Tables ,5 
and 6 is for initial break-in testing only. Pinal placement information 
after all retesting had been completed was not obtained. The boxed data 
show the results of the Center's reccpranendations with respect to the 
break- in level to be administered at each grade level • 

- ije initial break-in test level administered was accurate for 74 • 2% 
of the site type A learners across all grade and skill levels. Of the . 
griand total of learners who required retesting, 53.6% needed to be 
retested up a level and 46.4% down a level. The total retesting per- 
* centages by grade level for the site type A learners were Grade 1, 
28.2%; Grade 2, 26.8%; Grade 3, 35.3%; Grade 4, 35.0%; Grade 5, 19.5%; 
and Grade 6, 11.6% for a grand total retest percent of 25.8. 

Substantially more retesting was required in the site type B thap 
in the site type A schools. The initial break-in test level was accurate 
for just 65.6% of the site ty^p B learners across all grade and skill level 
Of the grand total of Itorners who required retesting, most (89.1%) needed 
to be retested down a level while only 10.9% needed to be retested up a 
level. The total retesting percentages by grade level for the site 
type B learners were Grade 1, 7.4%; Grade 44.8%; Grade 3, 33.3%; 
Grade 4, 44.6%; Grade 5, 36.7%; and Grade 6, 41.2% for a grand total 
retest percent of 34.4. 



Skill Instruction 

The dates for beginning skill instruction were scheduled to allow * 
sufficient time for the retesting activity just described. If events 
were on schedule, the field test term permitted an implementation period 
of nine months for the primary levels (Grades 1-3) and eight months for 
tl^e intermediate levels (Grades 4-6) before comparison data were collected 
fbr the implementation groups. Descriptive information about the skill 
instructional aspect of implementation is presented in Chapter III in 
the evaluation of f ield l^st objective 1, product implementation. 



INSTRUMENTATIOH AND METHOD 

The instrumentation and the data collection schedule for the field test 
objectives are^^^uranarized in Tables 7 and 8. Discussion will now focus 
on the individtfal objectives. 



Product Usability 

Objective 1 ; Product implementation . One measure of produfct 
usability concerned whether the materials Would yield implementation 
practices in accord with the developer's plan. A necessary first step 
was to determine whether the field test faculties were able to meet the 
requisites for effective implementation. The primary sources for 
descriptive information on product implementation were the interview 
guides administered during monitoring visits to the schools by Center 
personnel. The interviews^ides ( Appendices E, F, and G) , were conr 
s true ted primarily to obtain information specifically about the requi- 
sites for effective implem«^tion. ^ 
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TABLE 5, 

' BREAK-IN TES-ftNG SUMMARY FOR SdWOLS WHERE READING 
ACHIEVEMENT IS TYPICALLY AT OR ABOVE GRACE LEVEL (SITE TYPE A)* 







4 






T e s 


t L € 


- 

i V e 


1 Admin is t 


T" ^ d 




Test Level 


Grade 




A 




; B 






C 


D 




E 


F 






Placement 


Level 


N 


% 




N 


a 
% 


si 


% 


vt 


% 


N % 


N 


% 
























""^ 










Accurate 


1 


i473" 




8! 




• 


1 


xuu • u 














Test Up 




!l86 
1 


28,2ri 






0 


0.0 




\ 








Test Down 




1 " 










0 


0.0 














Total JRetest 




1186 


28. 


21 






0 


0.0 














Total N 




1 

[659 










.1 
















Accurate 


2 


18 


51. 


4 


j458 


74.2} 


4 


100 .0 














Test Up 




17 


48. 


6 


1 62 


10.01 


0 


0.0 














Tps t Down 










1 97 


15. 7[ 


0 


0.0 














Total Retest 




17 


48. 


6 


|l59 


25. 7| 


0 


0.0 














Total N 




35 






|6i7 


J 


4 
















LLL a i»C 


3 








97 


82.2 


[T34' 


60.9 


2 


66. 7 




















'8 


6.7 


|125 


22. 8 1 


1 


33.3 










T^'q^ Down 










13 


11.0 


1 89 


16.2 1 


0 


0.0 










Total Retest 










21 


17.7 


|214 


39.0 j 


1 


33. 3 










Total N 










J.18 




}548 


_ 1 


3 












\^ ' * ' a 


4 








6 


85.7 


79 


61.2 


1 359 


^ C At 

65 • 4 1 


4 100.0 








X ca i« 










1 


14.3 


4 


3.1 


|l44 


26. 2| 


0 0.0 








T&s t Down 


• 








0 


0.0 


46 


35.7 


1 46 


8.41 


.0 0.0 








Total Retest 










1 


14.3 


50 


38.8 


[190 


34.6 1 


0^ 0.0 








Total N 










7 




129 




i549_ 


- J 


4 " 








Accurate 


5 












3 


60.0 


47 


56.0 


}"500'"8372j 


29 


100 


.0 


Test Up 














0 


0.0 


18 


21.4 


1 1 


0 


0 


.0 


Test Down 














2 


40.0 


19 


22.6 


1100 16. 6j 


0 


0 


•0 


Total Retest 














2 


40.0 


37 


44.0 


liai 16.8 1 


0 


0 


-0 


Total N 














5 




84 




l4oi_ ! 


29 






Accurate 


6 
















5 


41.7 


7.7 '86.5 


|555~ 


""89 


Ts 


Test Up . 


















1 


8.3 


1 1.1 


1 1 


10 


.2 


Test Down 


















6 


50.0 


'11 12.4 


1 64 


.3 


^Total Retest 


















• 7 


58.3 


12 13.5 


1 65 


10 


.5 


Total N 
















12 




89 


1 620 







*N = 9 schools 
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TABLE 6 

BREAK-IN TESTING SUMMARY FOR SCHOOLS^ WHERE READING 
ACHiEVEHENT IS TVPICALLY BELOW GRAOS LEVEL (SITE TYPE B] * 







T e s 


t 


L e 


V e 1 
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t e r 
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Test Level 


Grade 




A 






B 




C 
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Placement 


Level 
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% 


N 
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Accxirate , 
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1 12 
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V^est Dovm 




1 


- 




















Total Retest 




I ^2 


7. 


















■ 


Total N 




lJL63^ 












A 










Acurate - 


2 


20 


95. 


2 


! 54 47.8 














Test Up 




1 


4. 


8 


1 0 
















Test Down^ 










1 59 


52.2 




\ 








• 


Total Retest 




1 


4. 


8 


1 


52.2 














Total N 




^ 21 






Q]L3_ 


1 














Accurate 


3 








83 


62.41 


35 


79.5 
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4 


3..o| 


1 


2.3 










Test Down 










46 


34.6 


Q 
O 












Total Retest 












37. 6[ 


9 


20.5 










Total N 










Ij33_ 




44 








r 






















V 








Accurate ' 


4 








12' 


46.2 


' 65 


56.01 


10 


62.5 


1 


100.0 


Test Up 










0 


0.0 


7 


6.0 


2 


12.5 


0 


0.0 


Test Down 










14 


53.8 


44 


37.9 


4 


25.0 


0 


0.0 


Total Retest 










14 


53.8 


1 _ 
1 51 


43. 9j 


6 


37.5 


0 


0.0 


Total N 










26 




1 116 


_l 


16 




. 1 




Accurate 


5 








3 


75.0 


67 


63.2 


T 








Test Up ^ 










0 


0.0 


0 


O.CJ 




15. 0| 






Test Down 










1 


25.0 


39 


36.8 


9 


22.51 






.Total Retest 










1 


25.0 


39 


36.8 


15 


37.5] 






Total N 










4 




106 


f 


40_ 


J 






Accurate 


6 












4 


36.4 


59 


59.6 


"4l 


~~6"i72"[ 


Test Up 














0 


0.0 


. 3 


3.0 


r 0 


o.oj 


Test Down 






V 








7 


63.6 


37 


37.4 


1 26 


38.81 


Total Retest 












7 


63.6 


40 


40.4 


26 


38. 8j 


Total N 














11 




99 






—- tV' 



























*N = 2 schools ^ 
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TABLE 7 

INSTRUMENTATION FOR THE FIELD TEST OBJECTIVES 



Field Test Objectives 

Product Usability 
1. ' In^lementation 



Instrumentation 



2. Feasibility 

3. Deficiencies 

Product Effectiveness 

4. Con5)€urative performance — 
program-embedded tests 

5. Comparative performance — 
standardized reading 
achievement tests 



6. Comparative performance of 
individual leaurners — 
standardized reading 
achievement tests 

7. Mastery after instruction 



8. Skill retention 



Second Visit Interview Guides: Principal 
or Reading Specialist, and Unit Leader 
or Teacher (Appendix E) 

Third Visit Interview Guides: Principal 
or Reading Specialist, and Unit Leader 
or Teacher (Appendix F) 

Fourth Visit Interview Guides: Principal 
or Reading Specialist, and Unit Leader or 
Teacher (Appendix G) 

Same as for (1) 

Comment Cards (Appendix H) 



Wisconsin Tests of Reading Skill Develop- 
ment: Cdipqprehension subtests 

Listening and Reading subtests of the 
Cooperative Primary Tests (1965) ; 
Reading, :Con>P"hens ion siabtest of the 
Comprehensive Testa of Basic Skills (1968) 
' ' ' ■ ^ 

Reading Comprehension svbtest of the 
Ccpiprehensive Tests of Basic SHiUs (1968) 



Postassessment Following Instruction Form 
(Appendix I) . 

Wisconsin Tests of Reading Skill Develop- 
men t : Compr ehens i on subtests 
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; TABLE 8 

DAm CG^LECTION. SCHEDULE FOfe THE FIEID TEST OBJECTIVES 



-J 

Dates 



Usability Objectives 
Procluct Imjplementatlon, 
Feafiflbillty, Deficiencies 

(1-3)*/^^ 



Effectiveness Objectives 
In^j^enentatlon/ Mast£ry 
Pr^pimplementation after- Skiai 

Comparison Instruction Retention 

^ (4-6)* (7)V _ (8)_* 



1973 

September 



October 



November 



December 

1974 
Janviary 

February 
March 

April 
•May 

September 

October 

November 



Monitor Installation 
(Visit #1, 7 schools) 



Monitor Implementation 
(Visit #2, 5 schools) 



Monitor Implementation 
(Visit #2, 5 schools) 



Monitor Implementation 
(Visit #3, 5 schools) 



Monitor Implementation 
(Visit #3, 5 schools) 



Monitor Implementatibh 
(Visit #4, 9 schools) 



Break-in testing, 
LeV^ls A-D; 

Baseline data, Gr. 1-3 
(Gr. 1-4 site type B) 



Break-in testing. 
Levels E-F; 

Baseline data, Gr. 4-6, 
(Gr. 5-6 site type B) 



r 



Re test 



i 



Posttest 
after 

In s true tion Re tes t 



\ 



Re test 



Collect 
Post- 
assessment 
Comparison data, Gr. 1-3 Following 
(Gr. 1-4 site type B) Instruction 

forms 



Comparison data, Gr. 4-6 
(Gr. 5-6 site type B) 



Re test 



^Numbers in parentheses refer to field test objective nianbers 
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Almost all schools received four monitoring visits. The first 
visit had" two purposes: to introduce the Center monitor to the field 
test faculties and to obtain Information about the local inservice. The 
second and third visits, conducted approximately fo\jr months apart at 
each school, focused on the specifics of product Impleroentation. The 
fourth was essentially a follow- through visit to ensure that ijnple- 
mentation was tjdhtinuing with the new school year. During visits 2, 3, 
and 4 the buildifng principal or reading specialist was interviewed in 
addition to several (usually three) teaching staff nt€mbers. Because of 
the impl^anentation schedule, primary level teachers were the focus for , 
the second visit, and intermediate level teachers were the focus for 
the third. Teacher/unit leader interviewees were selected by the reading 
specialists or building principals who were instructed that the inter- 
viewees should possess a high degree of product familiarity and repre- 
sent a range of viewpoints. . 

* Objective 2 ; Product feasibility . Feasibility of the Comprehension 
element for elementary schools is related to the degree to which field 
t^t faculties were able to meet the requisites for effective imple- 
mentation. The interview guides previously described were the major 
data source, then, for information about implementation practices for 
both objectives. Additionally, the guides sought feasibility infor- 
mation in terms of teacher reaction to the J^foaxlct as a whole and to its 
specific objectives. : ■ j *j ,j 

Objective 3 ; Product deficiencies . At the outset of the field 
test, schools were supplied with Comment Cards (Appendix I) upon which 
they were encouraged to record problems or comments about specific 
product. components. Although the primary function of the cards was to 
alert the developer to specific i^ro'blem areas, notations of success were 
also welcomed. The cards were collected during monitoring visits and 
were also periodically mailed to the Center from the schools.- 



Product Effectiveness 

Objectives 4 and 5 were designed to contrast the performance of 
learners of the same age/grade level in the field test sites in difT 
ferent academic years— immediately prior *o installation (preimple- 
mentation group) and near the end of the first year of implementation 
(implementation group) . Each age-grade level prein?)lementation group 
served as the control group for the group of learners one year younger. 

The instruments for measuring objectives 4, 5, and 6 comprised an 
evaluation testing program administered to the baseline, or prein?>le- 
mentation group, in September 1973 for grade levels 1 to 3 (1 to 4 in 
site type B schools) and in November 1973 for grade levels 4 to 6 (5 to 
6 in site type B schools) . The testing program was then administered to 
the comparison, or implementation group, one year later. All materials 
and scoring services associated with the testing program were provided 
by the Center; the field test faculties administered the tests. 

Objective 4 ; Comparative performance' — program-embedded tests . The 
Wisconsin Tests of Reading Skill Development : Comprehension subtests 
were the assessment instruments used to compare the mastery levels on 
the product's objectives of the Implementation group versus the pre- 
implementation group. The tests that were administered sampled from all 
of the Comprehension strands with the exception of Affixes . The Affix 
tests were not sampled because their value had been qdestioned. A 
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combination of usually three tests comprised a single test sitting. 
Table 9 lists the subtests used at each gradte level according to site j 
type. Table 2 should be consulted with respect to the skill levels and 
numbers listed. 

A con^uterized randomization program was used to assign students at 
eacK grade level to the test sittings.- The test groups were formed from 
school-provided class lists. A test sitting group was usually composed 
of 20 to 30 students remdomly selected from all the classrocnns or units in 

a : — 



TABLE 9 . 

PROGRAM-EMBEDDED TESTS USED FOR COMPARATIVE PERFORMANCE 
ACCORDING TO SITE TYPE AND GRACE LEVEL 



Sites llhere Typical Reading Achievement Was; 



At or Above Grade Level^ Below Grade Level 



Comprehens ion 
strands 


1 


Grade Levels 
2 3 4 


5 


6 


1 


Grade 
2 3 


Levels 
4 5 


6 




Main Idea 




Dl^ *D1 


ffil. 
El 






*B1^ 


*B1 


CI, *C1 
Dl 


Dl, 
El 




Sequence 


*A2 


D2 




F2, 
Gl 


*A2 






D2 


*D2 




Reasoning 


A4^ 
B3 


B3/ *C3 *D3 
C3 


*E3 • 


E3, 
G3 


A4, 
B3 


*B3 


A4, 
C3 


*C3 *C3 


*D3 




Detail 




*B4 C4, *D5 
*C5, 
D4 


D4, 
E4, 
F5 


*F5 




*B4 


*B4 


B4, *C4, 
C4, *C5 
*C5 


D4, 
E4 




Context 




D6, 
E5 












D6 






Affixes 




[ N 0 


T 


S A 


M P L 


E D ] 










Total skills 
Tested 


3 


3 5 6 
Total =28 


6 


5 


3 


3 


5 6 6 
Total = 29 


6 





fN = 9 schools, site type A 
N = 2 schools, site type B 
^Skill level and number 

♦Baseline data for these skills were confuted from a random sanple fy± each 
school's brfeak-in test data. 
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a school spanning a particular grade level. The number of test sittings 
at each grade level and a school's grade lev^l enrollment determined the. 
number of times a student^was tested. Studeni^s in schools with high 
enrollments were usxially assigned to just one ;t^st sitting in 1973 and 
1974; Students in schools with low enrollments were assigned to more 
than one test sitting to acccxranodate the needs of , the testing program. 

The field test design called for the same testing program to be 
administered in both 1973 and 1974, Inspection of the 1973 baseline 
data, however, indicated a need Cor alterations in the tests being 
assessed. For exanqple, the 1973 baseline perfprm«mce was so high for 
some ot the skills that gain scores were unlikely to result from the 
1974 testing. The 1974 testing program was subsequently revised to 
accommodate the needed alterations. Tests administered in the 1974, 
but not in the 1973, testing program are identified ^n Table 9 by an 
asterisk. 

Alfl of the 1974 comparison data were derived from students assigned 
to test sittings by the randomization procedure previously described. 
The 1973 baseline data were also collected by this method for 19 of the 
28 tests administered in the site type A schools and for 15 of tjie 29 
tests administered^ in th6 site *type B schools. 

The . 1973 instructional break-in test data became the baseline data 
for the 9 and 14 skills of the respective two site groups presented with 
asterisks in Table 9. Tests were selected from the skill levels at 
which the majority of the students were initially tested for break-in 
purposes at each grade level. (See Tables 5 and 6 for the majority 
break-in level for each grade level.) Data for randan samples of 25 
students who were administered the majority breaUc-in level for each ^ 
grade level have been reported as the 1973 baseline data for these skills 
presented with asterisks. This data source is not uniform, however, 
because of the variability from school to school in the percentages of 
students tested at the majority break-in level for each gr^de level. 
Thus, the baseline data for skills tested in the 1974, but not in the 
1973, testing program usually represent less than 100% of the students 
at a grade level. 

Objective 5 : Comparative performance — standardized reading achieve- 
ment tests . Siabtests from the Cooperative Primary Tests (CPT) (1965) 
and Cong?rehensive Tests of Basic Skills (CTBS) (1968) conqprised the 
standMdized reading achievement test portion of the 1973-74 evaluation 
testing program. These tests were used as the program-independent 
measure for comparing the iii4)lementation group with the pfeimplementation 
group. The tests administered at each grade level are identified in 
Table 10 according to site type. The tests represented the best match 
possible between the content of the product's objectives and reading 
comprehension subtests of staindardized tests. A standardized test was 
administered at each grade level with the exception 6f Grade 1 for the 
site type B schools. 

The student randomization procedtire described for objective 4 was 
also used for objective 5 to collect the 1973 baseline data at all grade 
levels and to collect the 1974 comparison data^ at grade levels 1 to 3 
for the site type A schools and at grade levels 1 to 4 for the site 
type B schools. The reason for the nonramdomization in 1974 for the' " 
CTBS test sittings at grade levels 4 to € and 5 to 6 for the two respec- 
tive site types will be explained in the next section. 
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TABLE 10 

I 

STANDARDIZED READING ACHIEVEMENT TESTS 
USED €0R COMPARATIVE PERFORMANCE ( 
ACCORDING TO SITE TYPE AND GRAQ6 LEVEL 



> Sites Where Typical Reading Achievement Was^ 

At or Above Grade Level Below Grade Level 
Standardized Tests ^ Grade Levels Grade Levels 

1 2 3 4 5 6, 1 2 3 4 5 6 



Listening Subtest, Form 12Br 




'* X 


CPT (1966) 






Reading Subtest, Form 12B, 


X 




CPT (1965) 






■% 

Reading Subtest, Form 23B, 




X 

♦ 


CPT (1965) 




• 


Reading Comprehension Subtest, 




X 


Form Q Level 1, CTBS_^(1968) 






Reading Con^rehensiOTT Subtest, 


XX 


X X 


Form Q Level 2, CTBS (1968) 


-^T 





?N = 9 schools, site type A 
N « 2 schools, site type B 
*N6 test administered . 
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ca>jective 6 ; Compeuratlve perfcrmnnce of Individual learners — 
standardized reading^hievttttent tests . C3bjective 5 involved a cross- 
sectional comparison of different groups; objective 6 involved a longi- 
tudinal comparison of the same students. Levels 1 and 2 of the CTBS 
Reading Comprehension stabtest, administered as part of the 1973-74 
evaluation te^t^g program, were used as the assessment^instiruments. 
Data for this objective were obtained at grade levels 4, 5, and 6 for 
site type A students and, at grade levels 5 and 6 for site type B stu- 
dents. Students who had been randomly assigned to the 1973 CTBS eval- 
uation test sittings at grade levels 3, 4, and 5 (site type A) and 4 and 
5 (site type B) were intentionally assigned to the 1974 CTBS sittings ^t 
grade levels 4, 5, and 6, and 5 and 6 respectively, for the two site 
types. " ■ 
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Ob j ective 7 : Percent masteirs after skill instruction . The developer 
recommended procedure for determining whethelr learners have mastered the 
skill (s) for which they received instruction is administration of the 
product's cfiter ion-referenced test(s) as postassessments. A learner is 
judged to be a skill master when she/he attains an 80% or higher score 
on^the postassqssment. To ascertain the perc(entage of learners mastering 
the product's objectives at an 80% level ifflnediately after instruction 
in those skills, field test faculties were provided with copies of the 
Postassessment Following Instruction Form (Appendix I) at the beginning 
of the field test. Teachers were requested tb provide specific infor- 
mation on the form — skill level and nuxnber, date assessed, number in 
skill grox^), and resulting masters and nonmasters— for each skill that 
they taught and for which a postassessment was given. The forms were 
collected from each school in May at tl^e end of the 1973-74 school year 
and use of them was not reinstituted- in September 1974 for the last 
thre6 nionths of the field test. Therefore, the data collected represent 
an implementation period of eight months for primary level teachers and 
five months for intermediate level teachers. 

Objective 8 ; Skill retention . The product's criterion-referenced 
tests were used to ascertain learner retention rates on the product's 
objectives. There was a minimum eight-week period between skill mastery 
and the retesting which wa» performed by Center personnel during the 
monitoring visits to each school. Two kinds of skill retention were 
assessed: retenti6n fr<* the break-in testing and retention after * 
instruction. Testing for break-in retention involved the retesting of 
students on skills mastered at their initial break-in level. Since 
focused instruction on skills already .mastered would not be warranted, 
breatk-in retention was intended as a measure of retention without 
intervening instruction. Because the Center had scored each school's 
break-in tests, the identification of students and skills mastered was 
readily available. Because of the schedule for the break-*in testing, 
beginning of instruction, and monitoring activities/ retesting for 
break-in retention was conducted during the November 1973 and January 1974 
monitoring visits only. 

Retesting for retention after instruction was conducted during the 
March and May 1974 monitoring visits. At the outset of the field test, 
teachers were requested to date theix^Comprehension ptipil profile cards 
so that a record would be available bf when learners became skill masters 
after instruction. Dates were to be entered by the specific skills 
printed on the cards. Groups for retesting were subsequently formed 
which either met or almost met the objective's criterion of a minimum 
eight-week intermission between mastery after instruction and retesting. 



36 



III 

EVALU AT I ON OF PRODUCT USABILITY 



OBJECTIVE 1: PRODUCT IMPLEMENTATION 

Descriptive information edDOut product in^lementation in this report 
has been organized by requisites for effective implementation and by the 
attempts of the field faculties to meet them. A summary evaluation ^ 
statement for each requisite is presented in Table 11. Requisites 1 to 
3 refer to product installation, requisites 4 to 10 refer to, the instruc- 
tional, assessment, and management aspects of impffementation* In the 
discussion which follows comprehensive information for each requisite is 
presented across the 10 field test sites. Variations in requisite 
attainment will be noted where significant with respect to the school 
variables— muitiunit/conventional organization, reading achievement 
level, and number of Design elements being implemented. 



Preset Installation 

Requisites 1 and 2 concern teacher inservice training; requisite 3 
concerns placement of the learner into the program. 

Requisite 1 : Local representatives of the field test 
schools must attend a one-day Center-sponsored inservice to 
acquire information regarding implementation of the product. ^ 

Approximately three representatives from each site attended the Center- 
sponsored inservice. See Chapter II for a description of the inservice 
and Appendix D for the inservice agenda. Results of the inservice 
evaluation were positive; a majority Of the attendees rated 9 of the 10 
sessions as "very useful," the highest category on a ;^efulness scale of 
1 to 4. Additionally, all of the principal/reading specialist interviewees 
indicated that the Center inservice had provided sufficient information 
and materials about the product for them to conduct theiVjocal inservice. 
The sufficiency of the Center-sponsored inservice, then, established 
for field test objective 1. / 

Requisite 2 : Within one month following the Center Inservice, r 
^ local scHool representatives must conduct inservice sessions for 
local faculties to pro(ride information and support for imple- 
menting the product. The agenda for the sessions should include^ 
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ATTAINMENT OF REQUISITES FOR EFniCPIVE^^ttflEMENTATION' 



.Requisites 



Results 



1, Attendance at Center-sponsored 
inservice 

2, Local inservice training 

{ 

3. Identification of learners' 
initial instructional sldll levels 

4. Instruction appropriate to learners' 
skill deficiencies 



5. . Instructional groups formed on 
basis of assessment information 



. 6. Skill instruction for a ninimuiit 
of two hours per week, (selection 
of a Center-suggested scheduling 
alternative) 

7. Postassessnent following instruction 

8. Regrouping of rionmasters 

8 ■ , : 

9. Up-to-date record keeping 

10. Completion of at least one skill 
level per grade level 

" <^ 
ERIC 



Satisfactory. 

Genei^ally satisfactory. Exception: Insufficient opportunities 
for keying local 'niaterials to the product's instructional ' 
objectives. ^> ' 

Generally satisfactory. Exceptions: (1) New stufents not 
always tested, within ^a month of entering school (2) Kinder- 
garten break-in testing occurred in only 2 of 10 schools. 

Generally sitisfactor^'for iraplementors of formal pattern. 
Unsatisfactory, for implementors of informal pattern in terms 
of frequency with which skill 'masters received instruction. I 

Satisfactory for implementors of formal pattern. Generally 
satisfactory for implementors of informal pattern in terms 
of teachers' awareness of their develbpmental reading group 
members' skill deficiencies and subsequent instruction. 

Satiil^ctoty for scheduling aspect. Number of minutes per 
' week spent on skill instruction in descending order: (1) Formal, 
implementation pattern-no rotation of Design elements; (2) Formal 
implementation pattern-rotation of Design elements; (3) Informal 
implementation pattern. 

satisfactory. 

Satisfactory for implementors of formal pattern. Unsatisfactory 
for implementors of informal pattern. 

sAsfactory. 

No fard evidence available. ■ A satisfactory result can bfe 
estimated with a high degree of confidence for implementors of 
the formal pattern and with less confidence for implementors ^ ■ 
of the informal pattern.. ; 
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a description of the product, strategies for scheduling insLtr^c- 
t ion,, and opportunities for keying local materials to the pro- 
duct's behavioral objectives. i,. 

All sites conducted a local inservice within a month and a half 
after the Center inservice. The sessions ranged in length from 1 1/2 to , 
4 hours and all agendas included a description of the product and strategies 
for schedulihg instruction. The sessions were not alike with respect to 
the amount of time ^provided for keying local materials to the product's 
behavioral objectives. The unavailability of riiaterials for skill levels B 
to G was a deficiency for both the Center-led and local inaervices. 
Once the materials for Levels E to G had become available r one school 
provided a full day of inservice for keying and another school a half day. 
Two schools prbvided approximately an hour for keying of materials 
during their inservice. For 6 of the 10 schools, no opportunity for 
keying local materials was provided during the local inservice. 

A majority of \ the teacher interviewees reported their local inservice 
as adequate. Suggestions they offered for improv^ent included provision 
for (1) time for keying materials, (2) preparation time, (3) ••hands-on" 
approach to materials prior to inservice, (4) more extensive inservice, 
and (5) several short, spaced-out inservices baffore and after the onset 
of instruction which would provide time for becoming familiar with the 
materials, keying, and the sharing of implementation problems and successes. 

Requisite 3 ; Initial break-in testing, must be conducted to 
identify initial instructional levels for learners in grades 
1 to 6 (kindergarten second semester). Retesting must be con- . 
ducted within three weeks at a higher or lower level for learners 
who were tested at an inappropriate break- in level. New learners 
must receive break-in testing within one month after entering 
school* 

To place learners at grade levels '1 to 6 into the program at the 
beginning of the field test, break-in tests were administered for Levels 
A to D in September and for Levels E and F in November. Retesting up or 
down a level, required when learners show mastery of all or none of the 
skills at the level first tested, was completed within 1 to 1 1/2 months 
after results of initial testing had been received by the sites. The 
percentage of students requiring retesting is reported by skill and 
grade level in Table 5 for the sites where typical reading achievement 
is at or above grade level and in Ted^le 6 for the sites v^ere typical 
reading achievement is below grade level. The large percentages of 
children who required retesting may account for the 1 to 1 1/2 month 
period rather than the three^eek period recommended in the requisite. 

Interviewees during the second and third monitoring visits stated 
that new enrollees received break-in testing within a "^month after 
entering school. However, on the fourth visit, conducted near the end 
of the field test period in October of the 1974-75 school year, itkwas 
found that this was true for only two-thirds of the schools. The other 
one-third had not yet admini|^red bresJc-in tests to new students. 

According to the requisite, kindergarten students were to receive 
break-in testing during the second semester of their first school year. 
This occurred at two of, the 10 schools with instruction subsequent to 
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the break-in testing. Of the remaining eight schools, the 1973-74 
kindergarteners recefved September break-in testing as 1974-75 grade 1 
learners in three schools and had not received testing in five schools 
by the time of the October monitoring visit. 



Instruction , Assessment , and Management 

Requisites 4 to 10 pertain to specific steps of the implementation 
cycle after teacher inservice and learner placement into the program 
have been accomplished. As the field test progressed, the use of two 
distinctly different implementation patterns emerged— a formal pattern 
and an informal pattern (author-assigned terms) . The major distinction « 
between the two patterns lies in the membership andg^tability of the 
groups to which the skill instaniction was addressed. Group membership 
for the formal pattern was determined by learners' mastery status on the 
skills taught during each instructional skill cycle. Skill groups were 
cOTqposed pri mar ily of nonmasters of those skills. Membership changed as 
often as new skill groups were formed. Group membership for the informal 
pattern, on the other hand, was prescribed by learners' overall devel- 
opmental reading needs. In other words. Design Comprehension skill 
instruction was incorporated as a ba?al, or developmental, reading group 
activity and the groups were relatively stable. 

The formal inqplementat ion pattern was predominant in the eight 
schools where typical reading achievement is at or above grade level. 
All the primary ^ level units in one multiunit school and a lower primary 
unit in another multiunit school were exceptions. The informal pattern 
was predominant in the two conventionally organized schools where 
typical reading achievement is below grade level. 

Information f6r requisites 4 to 10 is presented with reference to 
the two ixnplementation patterns, formal and informal. 

Requisite 4 ; The local faculty must demonstrate a commitment 
to the Comprehension element that ensures that every learner 
receives instruction appropriate to his skill deficiencies. 

Once a learner's instructional level has been identifed and his/her 
mastery status for eabh skill at that level has been assessed, instruc- 
tion should focus upon the skills initially not mastered. Because non- 
mastery is the major criterion for skill group membership, schools which 
in5>lemented the formal pattern can be said to have generally attained 
this requisite. Learners generally received instruction only in the 
skills they had not mastered. Of the interviewees in schools impl^entinq 
the formal pattern, 62,5% reported that only nonmasters of a skill received 
instruction in-^t skill. Occasionally learners Received instruction in 
skills they had>lready mastered. There was variety of reasons for this: 
they had barely achieved mastery at break-in; they needed skill^eview; 
they were not applying the skill; or instruction was not currently being 
offered f<«. ^ skill they ha^ not mastered. 

In s3i^>a^and units using the informal pattern, instruction was 
less targeted, primarily because skill mastery status was not the criterion 
for membership in a particular developmental reading group. Although 
learners received instruction in skills they had not mastered, they also 
received ifistruction in skills they had mastered and at a much higher 
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frequencyHdsan v*is true for formal skill group learners. Just 1.1% of 
the interviewees in5)leinenting the informal pattern indicated that only 
the nonmasters of a skill received instructioni 

Requisite 5. t Instructional groups must be. formed on the 
basis of assfessment information. 

Assessment information about each learner's skill development 
status is transferred to a pUpil profile card by notching holes open for 
skills that have been mastered and by leaving holes unnotched, or closed, 
for. skills hot yet mastered. Skewering of the holes on the cards identifies 
the masters and nonmasters for each skill and is the technique by which^ 
skill groups of nonmasters are formed. Transfer of assessment information 
to the profile cards and subsequent skevrering ofr, the cards is a continuous 
px^ss. Users df the formal in^jlementation pattern always formed 
iT^t;itriictionaKskill groups on the ba;sis of the profile cards, i.e., 
asfd'ssment information. , 

jUsers of^the informal pattern did not form instructional skill 
grdups on the basis of assessment information, as a learner's Design 
Comprehension skill status was- not the criterion for membership in a - 
particular developmental reading gro\«>. Pecisions about what skills 
needed to be taught, however, were not based solely on teacher judgment. 
Interviewees implementing the informal pattern indicated that they used 
profile carfi information to identify their groups' skill needs. Q 

Requisite 6 ; Instruction must be provided that is related 
to the skill deficiencies of the instructional groups. Two • 
hours per wg^k of skill instruction is the minimum time require- 
ment . 

Two hours per week of Coinprehension skill instruction was proposed 
as the minimum time requirement when the field test plan was first 
formulated. Vfhen inservice materials were prepared to offer teachers 
alternatives for scheduling instruction, it b^^ame clear that two hours, 
per week every week might not be feasible. ' 

Five plans were subsequently offered to the field test participants 
to help them schedule instruction. Each plan was an alternative for 
incorporating Design skill instruction and either basal or individualized 
reading instruction into the traditional daily- "reading" block. The 
plans differed in the division of time for skill and basal instipuction 
and in grouping procedures, that is, whether concurrent groups were 
formed for each Design skill area, or skill areas were rotated. It was 
decided that the requisite should be judged attained if the schools 
implemented any of the scheduling alternatives. 

Implementation of the same scheduling alternative by all the units 
or grade levels within a' school was not a field test requirement. Four 
schools did implement an alternative on a school-wide basis. Two of 
these schools used the informal implementation pattern and offered daily 
instruction in Design Comprehension skills; one offered daily Comprehension 
instruction in formal skill groups; and one, using the formal pattern, 
rotated Design elements for instruction—a two-week cycle of Word Attack 
was rotated with a two-week cycle of Comprehension in the school's 
primary 4Jnit, while two-week cycles of Study Skills and Comprehension 
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were rotated in its intermediate tinit. Six schools implemented multiple 
plans that incorporated rotation as well as nonrotation of skill cycles - 
CMoviously there was wide vcuriability in the scheduling of implementation, 
not only among schools, but also within schools. 

The niamber of Design elements a school was implementing was not a 
differentiating variable in the scheduling alternatives adopted. For 
example/^ of the foior schools that were is^lementing just Word Attack and 
Cbraprehension, "only cpe school offered instruction daily in bdth elements. 
Two schools rotated cycles of Word Attack emd Con^rehension in their 
prinwury units, but offered Comprehension instruction daily in the inter- 
mediate units- At the fourth school, thd primary unit rotated Compre- 
hension instruction with instruction in a content area, while its inter- 
mediate unit provided Comprehension skill instruction without rotation. 
Likewise, no single scheduling preference emerged among the six schools 
implementing Study Skills in addition to Word Attack and Comprehension. 
The four schools that adopted a scheduling alternative on a school-wide 
basis were not the same four schools that were in5>lementing just Word 
Attack and Cojjprehension. The six schools that implemented multiple 
scheduling alternatives were not the same six schools that were imple- 
menting Study Skills in addition to Word Attack and Comprehension. 

Observation^ had indicated that the presence of Study Skills as an 
implementation variable was not worth investigating for several reasons. 
The degree and pattern of Study Skills implementation varied markedly 
within and across the six schools. The commercial edition of Study 
Skills had just been published at the start of the Comprehension field 
test and it took some schools several months to change from the field 
test edition to the commercial edition because of the substantial 
revisions that had been made. Therefore, the Changeover 'created a 
prolonged break in Study Skills implementation for some, schools. Also, 
the content-oriented as well as skill-oriented approaches to Study 
Skills implementation provided flexibility in scheduling for instruction. 
Across the six schools implementing all three Design elements, there was 
no visible pattern relative to Study Skills implementation that differen- 
tiated those who rotated elements for instruction from those who did 
not. 

Although scheduling was characterized by wide variability, some 
generalizations C2ui be supported. Across the ten schools, the rotation 
of skill cycles for Word Attack and COTiprehens ion instruction was pre- 
domincuit at the primary grade levels along with a daily basal reading 
period. The schools and units implementing the informal pattern were 
exceptions. Here, Word Attack, Comprehension, £md basal instruction 
occurred daily without rotation. Among the schools implementing the ^ 
formal pattern, some used schedules that involved rotation of skill 
cycles and some did not. In schools implementing ^tudy Skills through a 
skill-oriented approach, there was a tendency for Comprehension instruc- 
tion to be rotated with Study Skills at the intermediate grade levels. 
Rotati.on of Word Attack and Comprehension instruction was seen less 
often at the intermediate levels because the majority of learners have 
completed Word Attack by the time they have reached that level. 

The average nimiber of minutes per week spent on Comprehension skill 
instruction in schools implementing the for^^l pattern was derived by 
multiplying the total number of days by the minutes per ^ay spent in a 
Comprehension skill cycle and dividing by the length of time between 
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regrouping for Comprehension instruction* The average number of minutes 
is presented in Table 12 for two groins of interviewees, those who did 
not rotate Design elements for instruction, and those wiip dld« :jj|^ter- 
viewees who did not rotate, regrouped for Conprehension instruction 
either every two weeks or three weeks and provided a weekly^ average of 
103/3 minutes of inistructlon* The primeury level interview data for this 
group were not analyzed because they were either incomplete or suspect* 
Ihtexyiewees who rotated Design elonents for instruction regrov^ed for 
Comprehension either every four weeks, every six weeks, or every nine 
weeks; the average number of minutes per week they spent on Compre- 
hension instruction was 47«7« The majority of interviewees for both 
groups indicated that they were usually teaching just one skill groiqp, '.^ 
that is, they did not divide their time among several grovqps. 

The data from 16 interviewees who iioplemented the ihformal pattern/' 
are incomplete, because the majority were unable to report either the 



TABLE 12 

AVERAGE NUMBER OF MINUOES PER WEEK INISRVIEWEES USING FORMAL 
IMPLEMENTJIitriON PATTERN SPENT ON COMPREHENSION SKILL INSTRUCTIC^ 
ACCORDING TO REGROUPING FREQUENCIES 



Interviewees 
N (Grade levels represented) 


2 weeks 


Regropping Frequencies 
3 weeks 4 weeks 6 weeks 9 weeks 


Nonrotation of Design 
Elements for Instruction 






5 (Intermediate) 

{ 

4 (Intermediate) 


109 .'O 
. (60-125)* 






96.2 
(75-140) 




Average: 


103* 3 minutes 



Rotation of Design 
Elements for Instruction ^ 

5 (3 Primary; 56*0 

2 Intermediate) " (35-75) 

6 (4 Primary; 43.7 

2 Intermediate) (21-70) 

1 (Primary) 30^0 

Average: 47.7 minutes \ 



^ Range 
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average nvinber of days spent on a particuleur skill, the number of minutes 
per day, or how often they began instruction in ^ new skill. The number 
of days reported ranged from 6 to 35, and the jpuaiber of minutes per day 
Design Comprehension instruction was qivea:Mo^^t^ developmental reading 
group ranged from 5 to 10. Therefore, a dleve^^iNinta reading group might 
have received a minimum of 25 or maximum 50 pinptes of Design compr^en- 
sion skill instruction each week. 

Requisite 7 : Learners must be postassessed following 
instruction in order to ascertain their mastery level. 

Interviewees of both implementation patterns reported that they 
always postassessed following^ instruction. Hie product's criterion- 
referenced tests were used as the assessment instruments. Postassess- 
ment was a scheduled activity that signaled the end of a' skill cycle for 
inplementors of the formal pattern. Postassessment for a skill was less 
scheduled for implementdirs of the informal pattern emd occurred when 
learners appeared to be ready for it. 

Requisite 8 ; Learners must be regrouped in a maimer that 

is consistent with their current skill needs. f>;( 

This requisite is related to several of the other requisites. '^plts 
attainment is dependent particulaurly upon attainment of requisites 5, 7^ 
and 9 which pertain respectively to skill groups formed on the basis of 
assessment information, postassessment following instruction, and up-to- 
date record keeping (see below for discussion of requisite 9). Requi- 
site 8 is also interrelated with instaniction targeted at learners' skill 
deficiencies (requisite 4). In^lementors of the formal pattern can be 
said to have satisfactorily attained requisite 8. They regrouped learners 
for instruction that was consistent with their ctirrent skill needs by 
postassessing after instruction and by forming skill groups on the basis 
of profile ceurds, i.e., assessment information. Formal inplementors 
regrouped for Comprehension ever^ two to nine weeks (see requisite 6) , 
and their skill groups were cco^sed primarily of nonnasterte (see requi- 
site 4) . 

In^lementors of the informal pattern did not regroup in order to 
provide instruction consistent with their learners' cunrent skill needs. 
Informal in5>lementors were cognizeuit of their developmental reading 
groi^> members' skill deficiencies, they postassessed after instruction, 
and they kept group members' profile cards \ip-to-date. However, masters 
as well as nonmasters frequently received instruction in a skill. That 
is, instruction was targeted toward areas where learners were proficient ' 
as well as where they were deficient. 

Requisite 8 is based on the assuir?>tion that regrouping on the basis 
of nonmastery is the most direct and efficient method of providing 
instruction focus^ on the current skill naeds of learners. Imple- 
mentors of the formal pattern can be said to have attained requisite 8 
while Implementors of the informal pattern cannot. 
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Requisite 9 ; Profile cards niust be kept up-to-date and in a 
fashion that ensures their utility in instructional decisions, 

•"if- • ' 
Interviewees of both implementation pattems^;£^ported that the 

Conprehension profile cards or other recordkeeping ifleviceSr such as wall 

charts, were kept up-to-date. 

Requisite 10 > New skill groups must be formed at a rate that 
ensures cooipletion of at' least one level per grade in a school 
year throughout the field test period. 

The most precise method of measuring this reqiiisite would have been 
to analyze the differences between the number of skills mastered upon 
entering the program arid the nunibef of skills mastered ait the end of the 
field test period for samples of learners frora the field test sites. 
• However, the magnitude of the ?iata-gathering needs for the entire field 
test precluded use of this method. A less precise estijuate was ol:g«tined 
by analysis of the nxmiber of skills atf each skill level and the regrouping 
frequencies cited 4n the description of requisite 6. 

The number of skills at each level follows: 

4 skills 

4 skills 

5 skills 

6 skills 
6 skills 
6 skills 
5 skills 

Learners at the primary level working at Levels A to C would have had to 
master an average of 4.33 skills to complete at least one level per 
grade during the field test period; leamners at the intermediate level 
working at Levels D to G would have had to master an average of 5.75 
skills. Using the regrouping frequencies from Table 12 and a 40-week 
period as a school year base, learners who were regroijped for Compre- 
hension every tw, three, four, six, and even nine weeks, would hav^ had 
the opportxinity to receive instruction for the average number of skills 
cited above. This analysis applies only to implementprs of the formal 
pattern. 

Comparable data are not availjji^le for implementors of the informal 
pattern. Interviewees were usually unable to report how often they 
began instruction in a new skill. Further, it is not known how of ten a 
learner received instruction in skills not mastered, as opposed to 
skilla mastered. It ceui only be surmised that leaumers of the informal 
pattern did complete at least one skill level during the field test 
period. 

Summary / ' 

Evaluative iiiformation about product implementation has been pre- 
sented in relation to the attainment status of field test sites for each 
of >the developer-recommended requisites for effective in?>lementation. 



Level A 
Level B 
Level C 

n^o^;;- ' Level D 

Level E 
Level F 
Level G 



A s\anmatry evaluation statement for each requisite was provided In 
Table 11. The first three requisites pertJiined to product installation 
and were assessed as satisfactorily attained, with the exception of 
provision of sufficient opportiyiities for the .keying of local materials 
to the product's instinictional objectivies and placement testings for new 
students and second semester kindergartetl-JLevel students . 

•Requisites 4 to 10 referred to the instructional, assessment, and 
management Aspects of implementation. Variability in the attainment of 
these requisites was described in relation to the formal and informal 
implementation patterns used by the field test faculties. For the for- 
mal pattern, instructional groups were formed on the basis of ccwramon 
skill needs and membership changed as often as new skill groups were 
formed. For the informal pattern, instructional groups were formed 
,on the basis of basal reader ability with the membership of these groups 
remaining relatively intact. Implementors of the formal pattern were 
^een as having attained requisites 4 to 9 satisfactorily. For imple- 
mentors of the informal pattern, the results were satisfactory for requi- 
sites 7 and 9, generally satisfact^ for. requisites 5 and 6, and un- 
satisfactory for requisites 4 and 8. Satisfactory attainment for both 
implementation patterns. for requisite 10 was based on an estimate rather 

than actual evidence. 

It can be concluded that the teacher's resource files, teacher's 
planning guide and Center-sponsored inservice provided siifficient 
information for, the field test faculties to Implement the pjc'oduct 
accordi^g to the reqpiisites for effective' Implementation. Whether 
the requisites are attained or not is dependent upon the implemen- 
tation pattern which school staff members elect to use rather than 
upon the sufficiency of the prodj/ct's component materials. 

OBJECTIVE 2: : PRODUCT FEASIBILITY 

'yhe evaluative information provided for objective 1 showed that the 
Design Comprehension element can be installed and implCT according 
to the developer's plan, which is one measure of product feasibility. 
Teacher acceptance and degree of enthusiasm are additional measures. 

During the third monitoring visit, each school's principal or 
reading specialist was asked to rate teacher reaction to the Design 
Con5)rehension program. Their ratings are presented in Table 13. From, 
Table 13 it is cle«u: that most of the staff at the two site types had a 
• positive reaction to the program. 

Also dt:^ing the third monitoring visit, the 40 teacher/unit leader 
interviewees were asked to provide, independently of the interview, 
their perceptions of the program by rating statements on a five-point 
scale from (1) "strongly disagree" to (5) "strongly agree." The 40 
interviewees represented 22% of the field test staff since the total 
number of teaching staff m^nbers across the ten schools was 181. The 40 
included 16 primaury level teachers and 24 intermediate level teachers, 
32 of whom were using the formal implementation pattern, and 8, the 
informal pattern. 
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TABLE T3 

PRINCIPAL/READING SPECIALIST RATINGS OP 
STAFF REACTIONS TO WDRSD: COMPREHENSION 



Reaction 



• Staff Negative Lukewana Positive Enthusiastic 

Site Type A ^ a 

Schools where typical Most 1 1 9 

reading achievement ^ 

is at or above Some - 2 - . ' 2 

grade level (N 9) * 

A few 2 1 - 1 



Site Type B 
Schools where typical 
reading achievement 
is below grade 
level (N = 2) 

*N = 9 here, rather than 8, becaxise one school system has separate buildings 
and administrators for its primary and intermediate level students, 

^A particular grade level at one school 



Most - - 2 

Some ^ 1 * 

A few - - " 



Table 14 presents the interviewees' degree of agreement, according 
to implementation patterns, for nine statements. Statement 1 refers to 
agreement with the selection of skills identified on the whole. State- 
ments 2 to 7 refer to the objectives in each of the six strands. When 
the percent of respondents for categories 4 and 5 are summed, it is 
clear that implementors of bot^i patterns agreed with statenjient 1, that 
the Design had correctly identified skills students need for reading 
comprehension. There was also a high degree of agreement among the 
interviewees with respect to the objectives that had been identified for 
the Main Idea and Context strands. The lower percent of interviewees 
agreeing with the Sequence, Reasoning, Detail, and Affixes objectives 
confirmed other field test data which indicated the need for revisions 
in those strands. 

Statements 8 and 9 were designed to measure degree of enthusiasm 
for the product and the product's perceived effectiveness. In suxmning 
the percent respondents for categories 4 and 5, 72% felt that spending 
extra time to teach the skills was worthwhile. Ratings on this measure 
of enthus4Asm were highest for the formal implementors; over half of the 
informal implementors chose the neutrsbl category for statement 8. These 
same proportions between the two groups were maintained for statement 9, 
but there was more spread across the rating categories. Slightly more 
than half of the formal implementors agreed that the Design enabled them 
to teach comprehension skills better than any other program they had 
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• / - TABLE 14 

THIRD MONITORING VISIT INTERVISWEES * DEGREE OF 
AGHBEMBNT WITH STATEMENTS ABOUT WDRSD: COMPREHENSION 




Implementation Re- ^ Mean % Strongly % Disagree % Neutral % Agree % Strongly 

Pattern* spondents Rating Disagree (1) (2) (3) (4) Agree (5) 

1. Thip Wisconsin Design h^s identified skills that students need for reading 
cx|piprehension . 

■% 

Format ^ ' 32 4.22 0.0 3*1 3.1'' 62.5 31.2 

Inforaiial; / 8 4.00 0.0 0.0. '12.5 7^.0 12.5 

Total?;; 40 4.17 0.0 2.5 5.0 65.0 2?. 5 

ry of the objectives in the Main Idea strcuid is essential for reading 
eHension . 

Formal 30 4.30 0.0 0.0 3:^3 63.3 V 33.3 

Informal 7 3.71 0.0 0.0 28.6 ^71.4 0.0 

Tbtal : 37 4.19 0.0 . 0.0 8.1 64.9 27.0 

3* Mastery of the objectives in the Sequence stratnd is essential for reading 

comprehension. ' 

Formal 30 3.83 0.0 6.8 16.7 63.3 13.3 

Informal 6 3.50 0.0 0.0 50.0 50.0 0.0 

To^hl 36 3.78 O.Q 5.5 22.2 61.1 ' 11.1 

4 . y; ^Mastery of the objectives in the Reasoning streuid is essential for reading 
I ■ comprehension. / 

F<|rmal 27 4.00 0.0 3.7 14.8 59.3 22.2 

Intonjial 8 3.62 0.0 0.0 37.5 62.5 0.0 

T<|tal 35 3.91 0.0 2.9 20.0 60.0 17.1 

5. - Mastery of the objectives in the Detail strand is essential for reading 

comprehension. , ^ 

Formal^ 
Informal 
Total 

6. Mastery of the objectives in the Context strand is essential for reading 
comprehension. t 

Formal 24 3.92 0.0 4.2 8.3 79.2 ^8.3 

Informal 8 4.00 0.0 0.0 12*5 75.0 15.5 

Total 32 3.94 0.0 3.1 9.4 78.1 9.4 



30 


3.93 


0.0 


0.0 


23.3 


60.0 


16.7 


7 


3.71 


0.0 


14.3 


14 ..3 


57.1 


14.3 


37 


3.89 ^ 


0.0 


2.7 


21.6 


59.4 


16.2 
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Table 14 (cent.) 



Iaple«nt.tlo„ H «e» » Strongly » Disagree »,N.utr.l % Agree » S^°"9ly 

Pattern t,spondents Rating Disagre e (1) (2) P) (4) Agree yp, 

7. Mastery of the objectives in the Affixes strand is essential for reading 
comprehension. 

„ , 04 '3 87 0.0 0.0 25.0 62.5 12.5 

Formal 24 J. a/ u.u -37 c on 

, „ , Q 17 0.0 0.0 62.5 37.5 u.u 

Informal 8 3. J/ u.u ,^ ^ , q 4 

Total 32 3.75 0.0 0.0 34.4 56.2 9.4 

8. Spending extra time to teach the Design 's Comprehension skills is worthwhile. 

informal 7 3.57 0.0 0.0 57.1 28.6 14.3 

Total 39 3.97 0.0 2.6 2.5.6 43.6 28.2 

9. The Wisconsin Design enables me to te'ach comprehension skills better than any 
^ . "other program I've used. 

Formal ° 29 3.38 3.4 20.7 20.7 44,8. 10.3 

T 8 3 12 0.0 12.5 62.5 .^5.0 ., 0.0 

Informal o j.a.^ ,„ „ na -, An <; l a t 

Total 37 3.32 2.7 18.9 29.7 , 40.5 j 8.1. , 

*The formal implementation pattern was predominant in the eight site type A schools, _ 
the informal implementation pattern was predominant in the two site type B schools. 



used, while more than half of the informal implementors chose the 
neutral rating for statement 9. Across all nine statements, a majority 
of the formal implementors rated in categories 4 and 5, and a majority 
of the informal implementors rated in categories 3 and 4. 

During the fourth monitoring visit, conducted near the end of the 
field test, the 30 teacher/unit leader interviewees were -asked how they 
felt about continuing with the program. Of the responses, 83.3% were 
positive, 10.0 neutjcal, and 6.7% negative. 

The number of interviewees for the third and fourth monitoring 
visits is small in comparison to the total number of school staff who 
were implementing the program. Nevertheless, the data for the formal 
implementors are generalizable since ^ajority of those Interviewees 
were from multiunit schools and were presenting a unit viewpoint, rather 
than an individual viewpoint. It can be concluded that school staff in 
general considered the Design 's Comprehension component a viable program. 
Staff members who used the formal implementation pattern were, on the 
whole, quite enthusiastic about the program. Reactions to the program 
for interviewees of the informal implementation pattern were more neutral. 
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OBJECTIVE 3: PRODUCT DEFICIENCIES 

Coranent Cards (Appendix H) filled out by the field test teachers 
alerted the developer to deficiencies in the product's n^erlals. 

Specific component materials receiving conments were the teacher's 
planning guide, teacher's resource files, and the criterion-referenced 
tests. There are separate teacher's planning guides for Word Attack, 
for Study Skills # and for Comprehension, and one guide for Self-Directed 
Interpretive, and Creative Reading* The need was expressed for a siij^le 
planning guide that would provide several specific examples of how to 
schedule for concxirrent inqplementation of all the Design elements. 

Although the teacher's resource files received many positive 
ccoBnents, some teachers felt that more variety was needed in the type of 
skill development activities offered. They advocated that activities be 
less teacher -directed, adaptable to independent work, less paper and 
pencil, and more "game-oriented." The inclusion of activities for skill 
application was also suggested. 

Tests most frequently cited as having problems were those in the 
Se<3[ii€5hc6j. fleafsoAing, Detail, euid^ Affix strands. -Teachers commented that 
thWmcmioi^ load was; tTO great for the Sequence tests at Levels A, B, and 
C jauid that the picture size was too small for .both the Seqiience and 
Reasoning tests at Levels A and B. Teachers also identified test items 
where the pictui:^ was confusing* Some teachers felt that there 

was not enbugh difference between the levels of the skills in a strand 
as, for exan^le, in the Sequence "Before or After" skills at Levels B 
and C, and Context at Levels D ^pd E.^ Some children were able to master 
a Detail skill at a higher leVei but not at a lower level, calling the 
leveling of the skills into question. Across tfie strands, teachers 
identified test items that were particularly corifusing^ taroublesome 
for their students. • 

Teachers also identified more general problem ^areas . Many felt 
that there was too great a gap between Levels B and C. That is, chil- 
dren who were able to rather rapidly move through the listening skills 
at Level A and B could not then handle the reading required at Level C. 
They also felt that the leap from listening skills at Levels A axid B to 
reading at Level C was too great. It was recommended that reading be 
required for some of the Level B skills as a transition to Level C. A 
lower voced^ulary level for Level C was also recommended. The other 
general problem area concerned the Reasoning strand. Both school 
administrators and teachers advised that more teacher-directed material 
was needed for the upper-level reasoning skills. Teachers did not have 
the background in reasoning they felt was requisite for teaching those 
skills. 
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' IV • ' 

' A ' ' EVALUATION OF PRODl^|^i||jLVgNESS 

Thefe^ degree of in?pleinentation--i€'the field test sites provides one 
context for evaluating the product etf fee tiveness data presented for ^ 
field test objectives 4 to 8. The observed implementation period pro- 
vides another context for evaluation of the effectiveness data, par-^ 
ticularly in relation to comparative performance on the program-embedded 
tests, objective 4. As originally designed, the field test term permitted 
an implementation period of nine months for the prlmaury levels (Grades 1 
to 3) and eight months for the intermediate levels (Grades 4 to 6) 
before comparison data were collected for the implementation groups. A 
majority of the piJbary level teachers began isplementation in November 
of 1973 J ^ut it idTquestionable whether actual instruction had gotten 
underway by Fall m97^ befpre the caanparison data were collected in late 
September, ^j^txeeifore , the actual iaiplementation period for the primary 
levels was approximately seven months. Intermediate level teachers in 
eight of the ten schools began instruction on schedule in Janueury 1974, 
and all but one school had started in the Pall by October 1974. The » 
actual inrolementat ion period for the intermediate levels before the 
comparisoa data were collected in late November, then, was also approx- 
imately seven months. 

Virtually no implementation occurred for Grade 1 before the com- 
f)arison data were collected. Second semester kindergarten iinplementation 
occurred in just two of the ten schools. Implementation of the year- 
round-school concept^ began in 1974 intone of these two schools, meaning 
that comprehension instruction at all grade levels began again in July 1974 
for two tracks of students and in October 1974 for the third track. 
Except for the year-rotind-school, the beginning and ending of school 
dates were"^ approximately alike for all the field test kites . 

The evaluation for objectives 4 and 5 was designed to contrast the 
performance of learners of the same atge/grade level at the field test 
sites in different academic years, immediately prior to installation 
(Fall 1973, preimplementation group) and near the end of the first year 
of implementation (Fall 1974, implementation group) . Each age/grade 
level^ preimplementation group served as the control group for the group 
of learners one year younger. The grade levels reported in the tables 
for objectives 4 and 5 identify the grade levels tested; the change 
scores reflect the results of implementation efforts between Fall 1973 
and 1974. For example, change scores reported for grade 5 show the 
results of 1973-74 implementation in grade 4 as well as any' implementation 
in grade 5 prior to the Fall 1974 testing. 
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OBJECTIVE 4: COMPARATIVE PERFORMANCE— PpOGRAM-EMBEDDED TESTS ' 

Results of comparisons on the program-embedded tests between the 
preijnplementatlon groups and the implementation groups are presented In 
Te^jle 15 for the site type A schools and in Table 16 for the site type B 
schools. Percent of mastery information was obtained by computing the 
percent of students tested who perfonq^^ at an 80% or higher level on 
the test. Included in the tables are the values for the independent 
sample t^ tests computed on the raw score change of each skill assessed. 

* ■ » 

<s ■ . ■ ■ 

Site Type A Comparative Performance ^ 

Of the 28 Desi-qn objectives assessed, .the implementation groups 
(Fall 1974 data) mastered 22 objectives, or 79%, at a higher level than 
the preimplementation groups (Fall 1973 data) • Eight of these 22 gains 
and 1 of the 6 losses involve skills for which the 1973 and 1974 data 
sources are not tiniform. See "Product Effectiveness," Chapter II, for 
a discussion of the data sources. 

The raw score changes were statistically significant for 9 of the 
22 gains and for 2 of the 6 skills for which there were losses. Data 
sources were not uniform for 3 of these 9 statistically signif icamfc 
gains and for 1 of the 2 significant losses,. Skills and grade levels 
for which gains were statistically significant were Dl, Grades 4 and 5; 
A2, Grade 1; D2, Grade 4; B3,, Grade 2; E3, Grade 6;'B4, Grade 2; and D6 
and E5f Grade 4. Statistically significant losses occurred for A4 at 
Grade 1 and E3 at Grade- 5. 

When the comparisons are analyzed by grade level clusters, 6, or 
54.5%, of the 11 comparisons for the primary levels (1 to 3) were positive; 
16, or 94.1%, of the 17 comparisons for the intermediate levels (4 to 6) , 
were positive. It should be recalled that in the evaluative information 
presented for product implementation, field test objective 1, it was found 
that intermediate level interviewees of the formal implementation pattern . 
spent a higher average number of minutes per f week in skill instaniction than 
^did primary level interviewees. / 

Site Type B Comparative Performance 

Of the 29 Design objectives' assessed in these two schools, the 
implementation groups mastered 21 objectives, or 72%, at a higher level 
than the preimplementation groups. When raw score changes are compared, 
gains occurred for 22, or 76%, of the skills. Eleven, or half, of the 
raw score gains and 3 of the 7 losses involve skills for which the 1973 
and 1974 data sources are not uniform. 

Raw score changes were statistically significant for 9 of the 22 
gains a^d'for 3 of the 7 losses. The 3 significant losses all occurred 
at grade level 1 in schools where there l>ad been no kindergsucten or 
grade 1 incplementation prior to the testing* These losses, then, cannot 
be attributed to program effect* ' Data sources were not uniform for 5 
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TABLE 15 



i 



COMPARATIVE PEFPORMANCE ON PFDGBAM-?HBEDDED TESTS 
OF SOTTS IN SCHOOLS WITH TYPICAL BEADING ACHIEVEMENT 
AT OR ABOVE GRADE LEVEL (SITE TYPE A)* 




' Comprehension Skills 
Vm IDEA STRAND 



Grade 
Level 



Raw score Percent Raw score Percent Raw score 
H mean masters N mean masters mean. 



Percent 
masters t 



Dl"Topic Sentences lli)) 


J 


iL\i 


7 ^3 


24 3 

All J 


248 


7.24 


28.6 


-.09 


4.3' 


-.217 


l)l--Topic Sentences (15) 


4 


221 


,9.86 


49.3' 


217 


12.14 


*1') "1 

73.7 






6 263*** 


Dl-Topic Sentences (15) 


5 


250 


12.24 


73.2 


236 




' M 1 




9.9 


2.894*** 


Bl-rMain Idea (13) 


5 


■ 250 


9.86 


50.8 


236 


10.11 


37.0 ^ 




6 8 

v.o 


1.008 


SEOUENCE STRAND ' 






















A2-First or Last Event (14) 


I 


225 


9.95 


39.1 


111 

227 








15.1 


3;225*** 


D2--Explicit Sequence (20)' . 


i 


222 


14.85 


10.4 


nil 

21? 


ic nn 




1 15 


51.8 


3. 194*** 


F2--Implicit Sequence (30) \ 


6 


227 - 


26.24 


80.6 


247 


26. 00 






4 0 


1.632 


Gl-lBiplicit Sequence (12)' 


6 


227 


7.24 


29. r 


'247 


- 7.'45 


28.3 


.21 




.795 


REASONING STRAND ' 






















A4-0utccnies (12) 


1 


258 


9.61 


60.5 


227 


8.71 


41.9 , 


-.90 


-18.6 


-5,294*** 


B3"0utccDies (12) 


1 , 


' 258 


, 6c55' 


■ 19.8, 


227 


6.53 


16.7 


-.42 


-3:.l 


-1.735, 


B3"Outcoiiies (12) 


2 


245 


8.59 


47.3 


255 


9.20 


49.8 


.61 


' ' 2.5 


2.877*** 


C3~Cau8e-Effec1' (22) 


, 2 


' 245' 


13.92 


23.7 


255 


14.02 


29.4' 


. .10 


' 5.7 


.257 


b 

ca-cause-Effect' (22) 


3 


225 


17.44 


60.0 , 


, 240 


17.64 


60.4 ' 


' .20 


. .4 


■ .533 , 


b 

DS-Making Deductions (20) 


4 


221 


17.39 ' 


Ifd 


218 


17.75 


80.7 


.36 , 


1.5 


1.323 


b 

E3-Nakin9 Deductions |24) 


5 


214 , 


19.75 


60^.] 


236 


18.93 


52.1 


-.82 


-8.6 


-2.546** 



(continued) , 
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Table 15 (cont.) 



Coinprebension Skills 



Grade 



Fall 1973 
Kaw score Percent 



Fall 1974 1973-74 Change 
Haw score Percent . Raw score Percent 





Level 


N 


mean 


masters 


if 


mean 


masters 


mean 


masters 


t 


E3"MakinQ Deductions (24) 


6 


228 


19.33 


54.4 


?44 


20.46 


68.4 


1.13 


14.0 


3.668*** 


G3— Induction (15J 


6 


,228 


11.40 


60.5 


244 


11.99 


66.8 


.59- 


6.3 


2.114 


DETAIN STRAND 






















b 

B4--0etail (201 


' 2 


■224 


15.49 


60.3 


255 


16.77 

1 r r 


71.8 

r X 1 V 


1 28 

Ac ttV 


11.5' 


5.079*** 


C4i»Detail (20) 




206 


15.41 


66.5 

VV 1 ^ 


240 


15.22 


67.1 

V r 1 X 


-.19 


.6 


-.421 


jj 

C5-Negatives (23) 


3 


225 


17.05 


46.2 


248 


17.07 


47.6 


.02 


1.4 


.046 


D4--Clauses (21) 


3 


206 


13.82 


29.1 


248 


13.'21 


' 23.4 


-.61^ 




•1.689 ■ 


D5W(21) 


4 


221 


15.78 


46.2 


217 


16.18 


' 55.8 


.40 


9.6 


1.201 


D4-Clauses (21) 


. 5 


240 


17.27 


66.7 


229 


17.73 . 


74.2 


.46 


7.5 


1.637 


E4-£iiibedded Parts (20) 


5 


240 


16.34 


' 67.9 


229 


16.66 


75.5 


.32 


7.6 


1.344 


FS-Eiiibedded Parts 
and Clauses (20) 


■ 5 


240 


14.60 


45.4 • 


229 


14.71 


48.5, 


.11 


3.1 


.338 


FS-Qnbedded Pajts 
and Clauses (2.0) 




rh 


16.48 


69.7 


247 


16.87 


74.1 


.39* 


4.4 


1.455 


CONTEXT STRAND ' 


i. 




















D6"Context (12) 


V 4 

y 


223 


9.29 


61.4 


218 


10.22 


74.8 


.93 


13.4 


3.940*** 


E5-Context (12) 


4 


223 


7! 32 


26,5 , 


218 


7.81 


24.3 


.49 


-2.2 


2.112** 
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Maximum possible score 
1973 and 1974 data sources, not uniform 
* N = 9 schools 



**p'< .05 

***p"< .01 
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TABLE 16 



COMPARATIVE PERFOP 
OF STUDENTS IN SCHOOLS WI 
BELOW grade' 



^CE ON. PA06RAH-EMBEDISD TESTS 
TYPICAL READING ACHIEVEMENT 
EL (SITE TYra B) 



> 

Cooprehension Skills 


Grade 
Level 


N 


Fall 1973 
Raw score Percent 
mean masters 


N 


Fall 1974 
Raw score Percent 
mean mastery 


^1973-74 Change . 
saw score Percent 
mean ' masters 


■4 ^ 


MAIN IDEA STRAND . y 


















# . 




•a b"* 
Bl— Paragraph Topics (15) 


2 


50. 


9.88 


44.0 


54 


9.63 


38.9 


-.25 


-5.1 


-.311 


a 

Bl—Paragraph Topics (15) 


3 


50 


,,10.84i 


52.0 


56 


12.02 


64.3 


1.14, 


12.3 


1.800 


CI— Paragraph Topics (15) 


4- 


35 


'11.34 


54.3 


49 


11.43; 


55.1 


.09" 


0.8 


.122 


Dl~Topic Sentences (15) 


4 


35 


, 5.03 


2.9 


49 


5.96 


14.3 


,93 


.11.4 


1.313 


a 

CI— Paragraph Topics (15) 


5 


50 


11.32 


56.0 


47 


12.98 


76.6* 


1.66 


20.6 


2.725*** 


Dl~Topic Sentences (15) 


6 


49 


9.16 


44.9 


49 


9.43 


34.7 


.27 


-10.2 


.318 


El— Main Idea (13) 


6 


49 


6.78 


10.2 


%' 


7.33 


14.3 


.55 


4.1 


.948 



SEQUENCE STRAND 
A2— First or Last Event^ (14) 
D2— Explicit Sequence (20) 
D2— Explicit Sequence^ (20) 
REASONING" STfiAND 



1 


50 


7.30 


12.0 


59 


5.39 


5.1 


-1.91 


5 


49 


10.88 


0.0 


47 


15.85 


61.7 


4.97 


6 


• 50 


.12.46 


30.0 


49 


13.82 


42.9 


1,36 



A4— Outcooes (12) 


1 


57 


8.74 


38.6 


59 


6.07 , 


5.1 


-2.67 


■ -33/5 


B3— Outcones (12) 


1 


57 


5.09 


1.8 


59 


3.02' 


1.7 


-2.07 V. 


-0-1, 

. 7.3 


B3— Outcomes* (12) 


2 


50 


4.68 


4.0 


53 


6.43 


11.3 


1.75 


A4— Outcomes (12) 


3 


54 


9.11 


51.9 


51 


.10.0(5 


72.5 


.95 


'20.6/ 


B3— putcones (12) 


3 


54 


6.26 


16.7 


51 




33.3 


1.94 


li5.6 



-6.9 ^3^321*** ' 
61.7 5.772V*' 
12.9 >^ 1.431. V 



I 



'.3.W7^**S^ 



1^. 
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Table 16 (cont.) 



Comprehension Skills 


Grade, 
Level 


N 


Fall 1973 
Raw score Percent 
mean masters 




Fall 1974 
Raw score Percent 
mean masters 


1973-74 Change 
Raw score Percent 
mean masters 


A. 
t 


C 3— Cause-Effect (22) 


3 


54 


10.11 


0.0 


31 


14.20 


29.4 


A An 

4.03 


^7. 4 


5 918*** 


a - » 

C3 — Cause-Effect (22) 


4 


50 


15. 16 




AQ 

43 


l3 . 0 J 


J4. / 


Al 


10 7 


.565 


C 3— Cause-Effect^ (22) 


5 


50 


14.48 


24.0 


50 


17. JO 


AO ft 

4o. U 


GO 


Z4. U 


A nn*** 

*■ . Ull 


D3 — Making Deductions^ (20) 


0 


50 


1 A AC 

14.40 


VlO ft 

4o. U 


40 




3U. \J 9 




2.0 


571 


CETAIL STKAND 






















o4--Detaii 12U/ 


"5 


3U 


11.60 


18.0 


J J 


11.21 


13.2 


-.39 


-4.8 


-.503 


B4r-Detail (20; 


3 




13.34 


38.0 


^C 
30 


14.6^ 


46.4 


1.27 


8.4 


.336 


B4— Detail (20) 


4 


39 


' 16.85 


69.2 


47 


16.77 


74.5 


-.08 


5.3 


-.126 


C4— Detail (20) 


4 


39 


13.33 


41.0 


47 


.14.32 


55.3 


.99 


14.3 


,860 


C5— Negatives^ (23) 


4 


50 


13.44 


16.0 


47 


14.74 


29.8 


1.30 


13.8 


1.385 


a- 

C4— Detail . (20) 


5 


50 


12.86 


32,0 


50 


15.76 


60.0 


2.9a 


28.0 


3.207*** 


C5— Negatives^ (23) 


5 


50 


13.28 


8.0 


47 


17.38 


48.9 


4.10 


40.9 


4.778*** 


D4-Clauses (21) ^ ^ 


6 


52 


14.90 


38,5 


46 


15.50 


34.8 


.60 


-3.7 


1.016 ■ 


E4— Embedded Parts (20) 


6 


52 


14.31 


42.3 


46 


14.24 


37.0 


-.07 


-5.3 


-.109 


CONTEXT STRAND 






















D€— Context (12) 


5 


46 


8.26 


34,8 


50 


^ 8.32 


36.0 


.06 


1.2 


.115 



^1973 and 1974 data sources. not, uniform. 
Maximum possible score 
Q ^ *N = 2 schools 
DU ** p < .05 

***p < .01 
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of the 9 significant gains and for 1 of the 3 significant losses. 
Skills and grade levels for which gains were statistically significant 
were CI, Grade 5; D2, Grade 5; B3, Grade 2; A4, Grade 3; B3, Grade 3; 
C3, Grade 3,- C3, Grade 5; C4 and C5, Grade 5. Statistically significant 
losses occurred for A2, A4, and B3 at Grade 1. 

Six, or 54.5%, of the 11 primary level raw score change comparisons 
were positive while 16, or 88.9%, of the 18 intermediate level con^arisons 
were positive. Information from school personnel about the average 
number of minutes spent per week in skill instruction was in5>recise and 
thus camnot be related to these comparisons. 

When ccmpaurisons are analyzed by Comprehension strands across the 
two site type groups, gains occiorred for 9 (82%) of the 11 Main Idea 
comparisons, for 6 (86%) of the 7 Sequence comparisons; for 13 (72%) of 
the 18 Reasoning comparisons; for 13 (72%) of the 18 Detail comparisons; 
and for all (100%) of the three Context comparisons. 



OBJECTIVE 5: COMPARATIVE PERFORMANCE— STANDARDIZED 
READING ACHIEVEMENT TESTS 

The Fall 1974 data soxarce for grade levels 4 to 6 (site type A) and 
5 to 6 (site type B) did not result from randcwi assignment of students 
to the test. In ortJer to measure field test objective 6, students who 
had been randomly assigned to the 1973 Comprehensive Tests of Basic 
Skills test sittings at grade levels 3, 4, and 5 (site type A) and 4 and 
5 (site' type B) were intentionally assigned to the 1974 CTBS sittings at 
grade levels 4, 5, and 6 and 5 and 6 respectively, for the two site type 
groups. The data, however, were uniformly derived from randomization 
procedures for the 1973 baseline data at all grade levels and for the 
1974 comparison data at grade levels 1 to 3 for the site type A schools 
and at grade levels 1 to 4 for the site type B schools.. 

Comparison data for the preinplementation and implementation groups 
in terms of raw score means eind normative information are presented in 
Table 17 for the site type A schools and in Table 18, for the site type B 
schools. Also provided are* values for the independent sample t tests 
that were performed on the raw score changes. 



Site Type A Comparative Perforgiance 

None of the six standardized test comparisons favored the preimple- 
raentation groups. Although kindergarten implementation occurred In just 
two of the nine schools, there was a raw score gain for the Listening 
subtest of the Cooperative Primary Test administered at Grade 1. Fall 
norms for this grade level are not available. The raw score gain for 
the CPT Reading subtest administered at Grade 2 was significant at the 
.01 level. 

The most dramatic increase on the Comprehensive Tests of Basic 
Skills occurred at Grade 6 where the raw score gain was significant at 
the .05 level. Changes in percentile rank occurred at Grades 4 and 6 
with increases of 4 and 7 respectively; there were no chamges in either 
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ME 17 



COMPARATIVE PERFORHANCE ON STANDARDIZED READING ACHIEVEMENT TESTS 
' OF STUDENTS IN SCHOOLS WITH TYPICAL READING ACHIEVEMENT 
AT OR ABOVE GRADE LEVEL (SITE TYPE A)* 



1^ 

0 









Pall 1973 






Fall 1974 ' 




1973-74 Change 


4;- 


Standardized Test 


Grade 




Raw score 








Raw score 




Raw score 










Level 


N 


' mean 




GE^ 




mean PR 


GE 


* mean 


PR 


GE 


t 


Listening Subtest, 


b 

r 


257 


32.80 




mm 


228 


33,76 -■ 


mm 


,96 


mm 




1.717 


Form 12B, CPT 






















< 




Reading Subtest, 


2 


244 


27.77 ' 


63 


m^ 


255 


30.25 63 


mm 




U 




2.910*** 


Pom 12B, CPT 


























Reading conprenension 


3 




25.70 


57 


jf J 




% dn 57 


3 3 


• /U 


u 


n 


.777 


Subtest, Form jj Level 1, 


























CTBS 

f 


























Rsaaing comprenension 


4 


ILi 


33.36 


Ars 

49 






1 

U 08 53 


4.2 


• U 




1 


.948 


Subtest, Form Q Level 1, 


























CTBS V 






t 




















Reading Comprehension 




243 


28.94 


54 


5.3 


234 


1 

29.24 54 


5.3 


.30 ' 


0 


.0 


.361 


Subtest, Form 2 Level 2, 






1 








*• 












CTBS 


























Reading Conprehension 




226 


31.46 


45 


5.7 


247 


33.30 52 


6.2 


1.82 


7 


.5 


2.500** 


Subtest, Form Q Level 2, 


























CTBS 



























^Entries for 'percentile ranks (PR) and grade equivalents (GE) are frco national norms tables 
Fall norms not available for grade level 1 

c-f: Grade equivalent at time of test administration: (c) 3.0; (d) 4.2; (e) 5.2; (f) 6.2 

AO *N s 9 schools \ 
00 Hp< .05 
***p"< .01 
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-TABLE 18 



COMPARATIVE PERFORMANCE ON STANDARDIZED READING ACHIEVEMENT lESTS 
OF STUDENTS IN SCHOOL^ WITH' TYPICAL READING ACHIEVEMENT 
BELOW GRAEE LEVEL (SITE TYPE B)*, 



Standardized Test 


drade 
Level 


N 


Fall 1973 
Raw score 
mean , 


PR^ 


(21* 


N 


Fall 1974 
Raw score 
mean PR 


' GE 


1973-74 Change 
Raw score 
mean PR GE 


t 


Listening Subtest, 
Forn 12B,tCPT 


2 


58 


32.59 ■ 


26 


mm 


53 


35.49 


37 




2.90 


,11 - 


2.270** 


Reading Subtest, 

Damn 0 ?Q HTifti \ - 

ton iSoi Uri 


3 


60 


24.20 


29 




56 


27.62 


40 


mm ■ 


3.42 


11 ~ 


2.785*** 


Reading Comprehension 
Subtest, Form Q Level 1, 
CTBS 




51^ 


21.84 


30 


3.0 


51 


21.92 


30 


3.0 


.08 


0 .0. 


.042 , 


Reading Comprehension 
Subtest, Form Q Level 2 
CTBS ■ 


5 


48 


19.10 


32 


3.7 

t 


52 


' 21.35 




4.0 


2.25 


5 .3 

4 


1.611 


Reading Comprehension 
Subtest, Form Q Level 2 
CTBS 


6^ 


49 


20.84 . 


25 


4.0 


48 


19.15 


20 


3.7 


-1.69 


-5 -.3 


-1.152 . 

1 

f 



Entries for percentile ranks (PR) for CPT are fran national norms, tables and for CTBS" are from large-city norms 
tables. Entries forbade equivalents (GE) are frcm national norms tables, 
b-d: Grade equivalent at time of test administration: (b) 4.0; (c) 5.2; (d) 6,2 
*N - 2 schools 

**p< .05 ~ . 

***?"< .01 . 
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percentile rsuiks or grade equivalents -at Grades 3 and 5, Grade 4 
experienced a grade equivalent gain of .1 and Grade 6, a gaia o^ .5. In 
terms of grade equivalent scores , Grade 4 was .1 and Grade 6 was .5 
below grade level in^^F^all 1973, while Grade 3 was .3 and Grade 5 was .1 
above grade level at that time. In Fall 1974 the grade equivalent 
scores were at level for Grades 4 aCnd 6 and again above level for Grades 
3 (.3) and 5 (.1> . 



Site Type B - Comparative Performamce 

Implementation groups achieved a higher raw score than preimple- 
•mintation groups on four of the five ccMnpar^sons; Raw scpre gains were 
'Statistically significant at the .05 level for the Listening subtest of 
the Qooperative Primary Test at Grade 2 and at the .01 level for CPT 
Reading subtest at Grade 3. There were percentile rank increases of 11 
for bQth C?T subtests. ^ At the. three grade levels administered the CTBS, 
no chemge in derived scores occurred at Grade 4 (CTBS Level 1) # a gain 
of 5 in percentile rank and .3 in grade equivalent occurred at Grade 5 
\ (CTBS Level 2) , and Grade 6 (CTBS Level 2) experienced a loss in both 
percentile rank (-5) and grade equivalent (-.3). ' 



OBJECTIVE 6: COMPARATIVE PERFORMANCE OF INDIVIDUAL 
LEARNERS— STANDARDIZED READING ACHIEVEMENT TESTS 

The CTBS test levels administered at each grade level must be con- 
sidered when emalyzing the longitudinal results for this objective. In 
the site type A schools, three grade level comparisons were performed: 

(1) grade level 3 with 4; (2) 4 with. 5; and (3) 5 with 6. Students were 
administered CTBS Level 1 as third and fourth graders. Level 1 as foxirth 
graders and Level 2 as fifth graders, and Level 2 as fifth and sixth 
graders. The grade level 4 with 5 comparison, then, involves a change 
from CTBS Level 1 to Level 2. Two grade level comparisons were per- 
formed in the site type B schools:. (1) grade level 4 with 5'; and 

(2) grade level 5 with 6. Students were administered CTBS Level 1 
fourth graders and Level 2 as fifth graders, and Level 2 as fifth 

^ixth graders. The grade level 4 with 5 comparison in the site typ^B 
schools, then, a\3o involves a chcuige from CTBS Level 1 to Levpl 2. 

^ In computing the re&ults for this objective it was not possible to 
use the grade equivalent gain scores from one grade level to the next 
given in Tables 17 and 18, because the year-to-year sampj.es were not 
strictly identical. The 1973 CTBS data in Tables 17 and 18 contain ; 
scores for several students who were no longer in attend£mce at those 
schools in 1974. Similarly, the 1974 CTBS data in both tables contain 
scores for students who laere not tested with CTOS in 1973, but were 
tested in 1974 to fill out a test sitting group. Fpr example, in the 
site type A grade level 3 with 4 comparison, 233 students were admin- 
istered the CTBS in Grade 3 in Fall 1973 and 229 students were admin- 
istered the CTBS in Grade 4 in Fall^l974. Of these students only 199 
were tested both years. ♦■'^ 
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For the other two site type A comparisons the following nximbers of 
students were administered the CTBS both years: 190 of the 223 students 
(Grade 4/ 1973) and 234 students (Grade 5, 1974)? and 221 bf'the 243* 
students (Grade 5, 1973) and 247 students (Grade 6, 1974). For the two 
site type' B compau:isons# the following numbers of, students were admin- 
istered the CTBS both years: 40 of the 51 students (Grade 4, 1973) and 
52, students (Grade 5, 1974) and 29 of the 48 students (Grade 5, 1973) 
and 48 students (Grade 6, 1974) • 

Frequency distributions of grade equivalent chemge scores were 
plotted for each of the grade level compeurisons using the data from only 
those students tested both years'. The distributions, by site type and 
grade level comparison, are shown in T€tble 19 in terms of the number and 
percentage of students for six grade equivalent change intervals. In 
the grade level 3 with 4 con^rison, 62/3% of the 199 students tested 
gained 1.1 or more years. A gain of 1.2 would be expected since the 
grade equivalent at the time of test administration in Grades 3 was 3.0 
and in Grade 4 was 4.2. Of the 190 students in the grade 4 compared 
with 5 comp2u:ison, 45.2% gained 1.1 or feaaceryealrs. In the grade 5 
cOTipared to 6 comparison, 46.6% of 221 stuiients gained 1.1 or more 
years. Across the three grade level comparisons for the site type A 
schools, 5.4% of the students experienced no change in grade equivalent 
scores, 27.9% a gain from a month to a year, 51.3% a gain of 1.1 or 
bdtter, while 15.4% experienced a loss in grade equivalent. 

For the site type B schools, 50% of the 40 grade level 4 with 5 
coi^arison students gained 1.1 or more, while only 17.2% of the 29 grade 
level 5 with 6 ccMtiparison students gained 1.1 or more yeaurs. Across the 
two site type B comparisons, 7.2% of the students experienced no change 
in grade equivalent scores, 33.3% a gain frcMn a month to a year, 36.2% a 
gain of 1.1 or better, while 23.2% experienced a loss in grade equivalent 

In order to test these data statistically, a standardized measure 
of an individual's relative stamding from year to year was necessary. 
Thi^ was accomplished' by utilizing the derived percentile ranks. The 
1973 percentile ramks of individuals were derived from tahle3 of national 
norms (large-city norms — site type B) on the basis of their CTBS raw 
score, test level, grade level, and time of year administered. Then, a 
projected raw score, based on the same^percentile rank fpr the next 
grade level, was determined. The differences between the 1974 obtained 
raw dcore and the 1974 projected raw score from the 1973 percentil^fe 
information were cOTiputed. These difference scores ^re then ^%|^n$ 
by grade level, using dependent t tests. The results are surmantj^^tn 
Table 19. The gains were greater thcui Would be expected for the sitf* 
type A grade level 3 with 4 and 5 with 6 ccxnpaurisons. For the grade 
level 5 with 6 site type B comparison, there w^s a more significant 
decrease than would be expected over a one-year period. 

OBJECTIVE 7: PERCENT MASTERS Apfe SKILL INSTRUCTIcfj 

Postassessment Following Instruction Forms (Appendix I) were com- 
pleted and returned to the Center hy the field test faculties of seven 
J of the eight site type A schools that were monitored an^ by one of the 
two site type B schools. Data stnnroaries from the forms cire provided in 
Tables 20 and 21 respectively for the two site typ^ groups. 
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TABLE 19 



0» 



DISMBUTION OF GRW3E EQUIVALENT CHANGES OF STUDENTS ADMINISTERED m CTBS 
READING COMPREHENSION SUBTEST IN BOTH 1973 AND 1974 / 



, Site Type 



.Adminis 
CTBS Level 



1973 1974 



tration / 
Grade Level Total 



1973 1974 



N 



Intervals of Grade Equivalent Changes 
-1.1 or -.1 to .1 to 1.1 to 2.0 or 

more -1,0 0 1.0 1.9 .more 

^ — ^ — 



N t N ^ N t N t K t N t 



Projected Vs. 
Actual Raw Score 
Differences 



t 



A-School8:''wlieH 
typical re^ii)9 
achievement is 
at or above 
grade level. 
(N = 9 schools) 
Total 

_s 



3- 4 



4 5 



5 6 



199 



190 



221 



610 



6 3.0 8 4.0 11 5.5 50 '25.1 64 '32.2 60 30.1 

15 7.8, 28jl4.7 4 2.1 57 30.0 40 21.0 46 24.2 

15 6.8 22 10.1 18 8.1 63 28.5 37 16.7 66 29.9 

36 5.9 58 9.5 33 5.4 170 27.9 141 23.1 172 28.2 



3.876" 



-.806 



3.515" 



B~Scho^ls where 
typical reading 
achievement is . 
below gra^e 
level, 

(N = 2 schools) 
Total 





4 5 

5 6 



40 
29 



0 0.0 4 10,0 ^ 2.5 15 37.5 14 35,0 6 15.0 
2 6.9 10 34.5„ 4 13.8 8 27.6 5 17.2 0 0.0 



.833 
-2.513* 
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2 2.9 14.20.3 5 1.2 23 33.3 19 27.5 6 8.7 



6^ 



r 



70 



»p < .05 
"d < J)1 

ERIC 
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TABLE 20 



PERCENT MASTERS AFTER SKILL INSTRUCTION IN SCHOOLS WITH 
TYPICAL READING ACHIEVEMENT AT OR ABOVE GRADE LEVEL 

(SITE TYPE A)* 



ERIC 





1 

Comprehension Skills 


' Number of 
Students 


Number of 
Masters 


Percent .v 
Masters 


MAIN IDEA STRAND 








Al-Picture topics 


290 


2o4 


91.0 1 


Bl- Paragraph topics 


524 


404 


77. 1 


Cl "Paragraph topics 


40 J 




OO . O 

i 


Dl -Topic sentences 


DO / 




Q Q a 


El-Main Idea 


282 


221 


"TO A 

/o. 4 


Fl-Main Idea 


301 


1 /b 


CO c 

bo. b 


Total 


2427 


1941 


80. 0 


SEQUENCE STRAND 








A2-First or last event 


492 


348 


70. 7 


B2-Before or after 


1095 


551 


50.3 


C2-Before or after 


657 


386 


56.0 


D2-Explicrit sequence 


422 


355 


84.1 


E2-Explicit sequence 


601 


271 


45.1 


F2-Implicit sequence 


1152 


139 


91.4 


Gl-Implicit sequence 


140 


118 


84. 3 


Total 


3559 


2168 


60. 9 


REASONING STRAND 








A3-Syn thesis 


191 


162 


84.8 


A4-0utCcmes 


395 


278 


70.4 


B3-Outcomes 


937 


656 


70.0 


C3-Cause-ef feet 


351 , 


310 


88.3 


D3-Making deductions 


187 


172 


92.0 


E3-Making deductions 


228 


175 


76.7 


F3-Complex deductions 


402 


165 


41.0 


F4-Inferring from principles 


278 


198 


71.2 


G2«Natural deductions 


35 


28 


80.0 


G3-Induction 


54 


44 


81.5 


Total 


3058 


2188 


71.5 




71 




(continued) 
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Tabid 20 (cont. ) 





Cooiprehension Skills 


Number of " 
S tudents 


Number of 
Masters 


Percent 
Masters 


DETAIL STRAND 








B4-Det:ail 


ceo 
558 


398 




C4-Derail 


238 


204 


85.7 


C5 -Negatives 


398 


314 


79.0 


D4*Clauses 


374 


287 


76.7 


D5"-Passives 


403 


312 


77.4 


E4-Einbedded parts 


72 


65 


' 90. 3 


FS-Embedded parts and clauses 


42 


39 


92.8 


Total 


2085 


1619 


77.6 


CONTEXT STRAND 








D6-Context 


236 


214 


90. 7 


E5-Context 


386 


281 


72.8 


G4-Context 


56 


54 


96.4 


Total 


678 


549 


81.0 


AFFIXES STRAND 








E6-Prefixes 


504 


357 


70.8 


F6-Suf fixes 


384 


309 


80.5 


G5-Pref ixes 


120 


82 


68.3 


Total 


1008 


748 


74»2 


Grand Total 


12815 


9213 


71.9 



*N = 7 schools 
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v:; -V- 



Mr 



* . . ' I • • • WITH 5ttf 3^'\i«BADIN«^':tekiitEM^^ LEtIteL .. 



' Cpft4>rehensipn Ska^ll^ 




Percent 
Masters 
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MAIN ^DEA S'tfwUiD 
V Al-Picture .topiic3 
Bl-Paragfaph topipg 
Cl-Paragraph topics 
Dl-Topic sentences ' 
El-Main idea 
Fl-Main idea 

Total 

SEQUENCE STRAND 

A2-First or last event 

B2 -Be fore or after 

C2 -Be fore or after 

D2-Explicit sequence 

E2-Explicit sequence 

F2-Implicit sequence 

G2-Implicit sequence 

Total »^ 
REASONING STRAND 

A3-Synthesis 

A4 -Outcomes 

B 3 -Outcomes 

C3-Ca use-effect 

D3-Making deductions 

E3-Making deductions 

F3-Complex deductions 

f4-Inf erring from. principles 

G2-Natural deductions 

G3-Induction 

Total 



.••7 . . ■ 

81 
112 
100 



391 

86 
60 
146 
190 
73 



555 



74 

85 
65 
128 
72 
75 



48 

i27 
72 
72 

ex 

280 

35 
20 
82 
115 
20 



272 



43 
48 
37 
83 
49 
49 



73.8 
81.8 
88.9 
64.3 
61.0 

71.6 

40.7 
33. 3 
56.2 
61.6 
27.4 



49.0 



58.1 
56.5 
56.9 
64.8 
68.1 
65.3 



499 



309 



61.9 



73 



^ (continued) 
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Table '21 (cont.) 



\ 







Number of 


Number of 


Percent 


Comprehension Skills 


Students 


Masters 


Masters 


DETAIL STRAND 






* 


' B4-Detail 


55 


51 


92.7 


C4-Detail ^ 


115 


83 


72.2 


C5-Negatives 


101 . 


66 


65.3 


D4-Clauses 


143 


67 


46.8 


D5-Passives 


, 109 . 


74 


67,9 


E4-Exnbedded paurts 


36 


29 


80.5 


F5 -Embedded parts and clauses 


— 


— 


— — 


Total 


559 


370 - 


66.2 


CONTEXT STRAND 








D6-Context 


113 


79 


69.9 


E5-Context 


31 


13 


41.9 


G4 -Con text 


— 


— 


- T 


Total 


144 


92 


63.9 


AFFIXES STRAND 








, E6-Prefixes 


58 


8 


13.8 


F6-Siif fixes 


— 


~ 


— 


G5— Prefixes 








Total 


58 


8 


13.8 


Grand Total 


2206 


1331 


60.3 










*N = 1 school 




















■If- ■ 
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ng order of total percent masters fdt the 
site type A schools, were Context (81.0%) , Main Idea (8d.O) , Detail 
(77.6), Affixes (74.2), Reasoning (71.5), and Sequence (60.9). The 
grand total percent masters across all skills and strands was 71.9%. 
Each of 36 Level A to G skills had been offered for instruction in 
the site type A schools, while the 25 skills encompassing Levels A to E 
had been offered for instruction in the site type B school.. The strands, 
in descending order of total'percent masters for the site type B school, 
were Blain Idea (71.6%), Detail (66.2), Context (63.9), Reasoning (61.9), 
Sequence (49.0), and Affixes (13.8). The grand total percent masters 
was 60.3%. 

OBJECTIVES: SKILL RETENTION ^ 

Two kinds of skill retention were assessed for Objective 8: re- 
tention from the instructional break-in testing and retention after - 
instruction. Testing for break-in retention involved the retesting of 
students on skills they had mastered at their initial breaJc-ln test 
level » S£nc4 focused instruction on skills already mastered is not 
warranted, Ipreak-in retention was intended as a measure of retention 
without intervening focused skill instruction. Retention after instruc- 
tion, on the other hand, involved the retesting of students on skills 
they had not mastered at break- in, but had mastered after r^eiving 
focused instruction. There was an intermission of almost eiglvt weeks 
(56 days) or more between the time mastery had been assessed either at 
breaU^-in or postassessment after instruction, and the Center retesting. 

The data for the two kinds of skill retention in terms of number of 
students retested and percent retaining mastery are presented by strand 
and skill in Table 22 for the site type A schools and in Table 23 for 
the site type B schools. Several factors contributed to the failure to 
collect a more comprehensive data set for this objective: (1) the fixed 
monitoring visit schedule, (2) percent of students who mastei^d specific 
skills at break- in, (3) the eight-week time interval needed between 
mcistery and retesting, (4) failure of some schools to enter dates on 
profile cards when skills were mastered, (5) alternating of Design skill 
areas for instruction, and (6) January 1974 startup for the intermediate 
grade levels. 



Site Type A Schools 

The strands, in descending order of retention of skills mastered at 
break-in, were Main Idea (92.9%), Detail (92.1), Reasoning (91.1), 
Sequence (89.1), and Context (87.5), for a grand total break-in retention 
rate of 90.9%. Affix skills were not retested. Students retained ^-tf' 
mastery after instruction for skills in the following strand order: 
Detail (93.3%), Main Idea (77.2), Sequence (74.3), Reasoning (71.5), and 
Affixes (50.0), for a grand total mastery after instruction retention 
rate of 73.0%. Context skills were not retested and there was little 



TABLE 22 



RETENTION OF SKILLS MASTERED AT BREAK-IN AND AFTER 
INSTRUCTION IN SCHOOLS WITH TYPICAL READING ACHIEVEMENT 
AT OR ABOVE GRADE LEVEL (SITE TYPE A)* 



Mastered at BreaJc-in Mastered after Instruction 



^, . , , Number of Percent 
comprehension Skills students Retaining 

Retested** Mastery*** 



Number of 

Students 

Retested** 



Percent 

Retaining 

Mastery*** 



Total 
"Number of Percent 
Students Retaining 
Retested Mastery* ** 



MAIN IDEA STRAND 
Al -Picture topics 
Bl-Paragraph topics 
Cl-Paragraph topics 
Dl-Topic sentences 
El-Main 'idea 
Fl-Main idea 

Total 
SEQUENCE STRAND 
A2 -First or last event 
B2 -Be fore or- after 
C2-Before or after 
D2-Explicit sequence 
E2-Explicit sequence 
F2-Implicit sequence 
Gl-In^licit sequence 

Total 
REASONING STRAND 
A3-Syn thesis 
A4«p&tcoraes 
BB-Out^mes 
C3-Cause-e£fect 
D3 -Making deductions 
E3-Making deductions 
F3-Corfi|jlex deductions 



25 
«20 
29 
27 
26 
127 



185 



100.0 
90.0 

100.0 
96.3 
76.9 
92.9 



J 
20 
17 
44 



89.1 



109 




85.7 
70.0 
82.3 
77.2 



74.3 



81.8 
85.7 



80.9 
37.8 



25 
20 
36 
47 
43 
171 



294 

21 
57 
67 
48 
49 
44 
37 



^ ipo.o 

9^.0 
97.2 
85.1 
79.1 
88.9 



43 


93.0 


11 


81.8 


54 


90.7 


49 


83.6 


27 ' 


74.1 


76 


80.2 


56 


83.9 


* 32 


71.9 


88 


79.5 


25 


100.0 


16 


100.0" 


41 


100.0 






23 


56.5 , 


23 


56.5 


12 


100.0 






12 


100.0 



83.6 

100.0 
77.2 
86.5 
95.8 
97.9 
90.9 
37.8 



(continued) 
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Table 22 (cont.) 



Mastered at Break- in 



Maetered after Inetruction 
Number of Percent 
Students Retaining 
p>^>.^^^** M^^iuirv*** Retested** Mastery*** 



Total 
Number of Percent 
students Retaining 
Rstested Mastery** * 



REASONING (cont.) 

F4-Inferring from 
principles 

G2 -Natural deductions 

G3-Induction 

Total 

DfiTAIL STRAND 

B4 -Detail 

C4-Detail 

C5-Negatives #^ 

D4 -Clauses 

D5-Pa80ives 

E4-Embedded parts 

F5-Embeddod parts 
and clauses 



Total 



CONTEXT STRAND 
D6-Context 
E5-Cpntext 
G4 -Con text 



Total 



AFFIXES STRAND 
E6-Pre fixes 
F6 -Suffixes 
GS-Prefixes 



23 



202 

25 

52 
24 
2S 
27 
12 

165 

25 
23 

48 



69.5 



91.1 

80.0 

96.1 
91.7 
88.0 
100.0 
91.6 

92.1 

80.0 
95.6 

8.7.5 



144 



15 



15 



Total 



18 



18 




71.5 



93.3 



93.3 



50.0 



50.0 



23 



346 



25 



52 
39 
25 
27 
12 



180 

25 
23 

48 

18 

18 



69.5 



82.9 

80.0 

96.1 
92.3 
88.0 
100.0 0 
91.6 

92.2 

80.0 
95.6 

87.5 

50.0 



50.0 



Grand Total 727 90.9 330 73.0 1057 85.3 



* N * 8 schools 

There was an intermiBsion of at least eight weeks between tHV\time of break-in 
testing or postassessment after instruction testingi and Center retesting. 

Mastery criterion! 80% 
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TABLE 23 



RETENTION OF SKILLS MASTERED AT BREAK-IN AND AFTER 
INSTRUCTION IN SCHOOLS WITH TYPICAL READING AOIIEVEMENT 
BELOW GRADE LEVEL (SITE TYPE B) * 



Mastered at Break-in Mastered after Instruction Total 
Number of Percent Number of Percent Number of Percent 
Comprehension Skills g^udents Retaining Students Retaining Students Retaining 
Retested** Mastery*** Retested** Mastery*** Retested Mastery *** 



MAIN IDEA STRAND 
Al-Picture topics 

Bl-Paragraph topics 41 87.8 

Cl-Paragraph topics 20 95.0 

Dl-Topic sentences — — 

El-Main idea 

Fl-Main idea — 

Total -t\ 61 90.2 
SEQUENCE STRAND 
^2-First or last event — . 
B2-Before or after 
C2-Before or after 
D2-Explicitr sequence — 
E2 -Explicit sequence — " 

F2-Implicit sequence 
Gl-Implicit sequence 

Total 
REASONING STRAND 

A3-Synthesis 27 88.9 

A4 -Outcomes 27 48.1 

B 3 -Outcomes 

C3-Cause-ef feet — ♦ 

D3-Making deductions . 



24 
22 

46 
24 

17 

23 



64 



28 

20 



75.0 
77.3 

76.1 

62.5 

29.4 
82.6 



60.9 



89.3 
70.0 



65 

20 
22 



107 

24 

17 
23 



64 

h 

27 
28 
20 



J 

83.1 
95.0 
77.3 



84.1 

62.5 

29.4 
82.6 



60.9 

88.9 
48.1 
89.3 
70.0 



(continued) 
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Table 23 (cont.) 



\ ' Mastered at Break-in Mastered after Instruction Total 

Comprehension Skills Number of Percent 'Number of Percent ^nnber of ^^^^^"^ 

, _ . students -Retaining Students ^tairving Students Retaining- 

Retested** Mastery*** Retested** Mastery*** Retested Mastery*** 

REASONING (cont.) . , 

E3^Making deductions — - ~" ~" 

F3-Ccmplex deductions — — — \ . 

F4-Inferring frc3in 
principles 

G2-Natural deductions — — — ■ 

G3-Induction — . — ; 



Total 



DETAIL STRAND 

B4-Detail 

C4-Detail, 

C5-Negatives 

D4-Clauses 

D5 -Passives 

E4r£mbedded parts 

F5-Embedded parts 
and clauses 



Total 



CONTEXT STRAND 
D6-Context 
E 5 -Con text 
G4-Context 

AFFIXES STRAND 
E6-Pre fixes 
F6-Suf fixes 
G5-Pref ixes 



Total 



54 

21 
37 



58 



68.5 

95.2 
78.4 



48 



20 



81.2 



85.0 



102 

. 41 
37 



74.5 

90.2 
78,4 



84.5 



20 



85.0 



78 



84.6 



Total 



Grand Total 



173 



81.5 



178 



73.0 



351 



77.2 



* N ^ 2 schools ^ . 

** There was an intermission of at least eight' weeks between the time of break-in 

testing or postassessment after instruction testing, and Center retesting. 
*** Mastery criterion: 80% 
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retesting for the Detail and Affix skills. Across both mastery at 
break-in and mastery after instruction, the retention rates in descending 
order were Detail (92.2%), Main idea (88.9), Context (87.5), Sequence ; 
(83.6), Reasoning (82.9), and Affixes (50.0), for a grand to,tal retexv.tion 

,..£a^^„.of _85.,3%._, , ■ — 

~ The meam ntmiber of days between mastery and retesting was 96.0 (the. ^ 
range was 63 to 146 days) for skills mastered at break-in and 87, T (the 
range ^as 48 to 165 days) for skills mastered after instruction. 

Site Type B Schools 

The mastery at break-in retention rates by stramd for these two 
schools were Main Idea (90.2%), Detail (84.5), and Reasoning (68.5), for 
a grand total retention rate of 81.5%. Ra|Mting of Sequence, Context, 
and Affix skills mastered at^reak-in did^ot occur. Retention rates for 
skills mastered after instruction were Detail (85.0% — one skill). Reasoning 
(81.2), Main Idea (76.1) and Sequence (60.9), for a grand total retention 
rate of 73.0%. There was no retesting for Context and Affix skills. 
Across both mastery at break-in and mastery after instruction, the 
retention rates in descending order were Detail (84.6%), Main Idea 
(84.1), Reasoning (74.5) , and Sequence (60.9), for a grand total retention 
rate of 77.2%. The mean number of days between mastery and retestfllg^ 
was 68.9 (the range was 68 to 70 days) for skills mastered at break- in 
and 77.2 (the range was 63 to 122 days) for skills mastered after inst 
tion. ^ 

Main Idea and Detail skills hiad the best retention rates for both 
site type groups. Also, skills mastered at break-in had better retention 
rates than did skills mastered after instruction. The mastery after 
instruction retention rate for both site type groups was 73.0%. 



SUMMARY 

Field test objectives 4 to 8 were concerned with product effective- 
ness. Data reported were separated according to reading achievement 
level which was typically at or above grade in eight schools (site 
type A) amd typically below grade level in two schools (site type B) . 

The evaluation, design for objectives 4 and 5 utilized a cross- 
sectional compeorison of the performance of learners of the same age/grade 
level in the field test sites in different academic years — ininediately 
prior to installation (Fall 1973, preiii5>lementation group) and near the 
end of the first year of implementation (Fall 1974, implementation 
group) . Each age/grade level preimplementation group served as the 
control for the group of learners at the saune age/grade level one year ^ 
later. 

Objective 4 pertaine(j| to comparative performance on the pre "July's 
program- embedded tests. In the site type A schools, comparisons wfte 
positive for 22,. or 79%, of the 28 Design objectives assessed*^ The raw 
score changes between the preimplementation and implementation groi|ps 
were statistically significant for 9 of the 22 gains and for 2 of the 6 
skills for which there were losses. Six, or 54.5%, of the 11 comparisons 
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for the primary levels (Grades 1 to\3) were positive while 16, or 94.1%, 
of the 17 comparisons for the intermediate levels (Grades 4 to 6) were 
positive. Comparisons favored the implementation groups in the site 
type B schools on 22!, or 76%, of the 29 skills assessed • Raw score 
changes were-statirStioally signif ixrant- f or-9 of for 3 
of the 7 losses. Six, or 54,5%, of the 11 primairy level raw score 
change compstrisons were positive while 16, or 88. 9<, of the 18 inter- 
mediate level comparisons were positive. 

' Objective 5 was concerned with comparative performance on 
standardized reading achievement tests. The Listening and Reading 
s^ubtests from the Cooperative Primary Tests (CPT) and Levels 1 and 2 of 
the Reading Comprehension subtest frcxn the Comprehensive Tests of Basic 
Skills (CTBS) were used as the program-independent measure for comparing 
the implementation group with the preimplementation group. All of the 
six standardized test comparisons in the site type A schools favored the 
implementation groups. The raw* score gain for the CPT Reading subtest 
administered at Grade 2 was significant at the .01 level, while gains in 
raw score means on the CTBS were achieved at Grades 3 to 6. Grade 4 
experienced a grade equivalent gain of .1 and Grade 6 a gain of .5, with 
corresponding gains in percentile rank scores of 4 and 7, respectively. 
There were no chanq/^B in either grade equivalents or percentile reuiks at 
Grades 3 and 5. The raw score gain at Grade 6 was significant at the 
.05 level. 

Implementation groups achieved a higher raw score than prein^le- 
mentation groups on fotir of the five comparisons for -objective 5 in the 
site type B schools. The raw score gains for the CPT "subtests were 
statistically significant at the .05 level at Grade 2>and at the .01 
level at Grade 3. For the CTBS administered at Grades 4, 5, and 6, no 
change in derived scores occurred at Grade 4, a gain of 5 in percentile 
rank and .3 in grade equivalent occurred at Grade^S, whileiterade 6 
experienced a loss in both percentile rank (-5) and grade equivalent .* 
(-.3). 

Objective 6 involved a longitudinal comparison of students who were 
administered the CTBS irt both 1973 and 1974 (610 students in the site 
type A schools; 69 students in the site type B schools). Across the 
three grade level con^arisons (grade level. 3 with 4, 4 with 5, and 5 
with 6) for the site type A schools, 5.4% of the students experienced ilo 
change in grade equivalent scores, 27.9% a gain from a month to a year,, ^ 
51.3% a gain of 1.1 or better, while 15.4% experienced a loss in grade 
equivalent # In terms of projected versus actual raw score differences, 
the gains vere greater than would be expected for the grade level 3 with 
4 and 5 with 6 comparisons (significance level of .01 for both compar- / 
isons) . Across the two site type B comparisons (grade level 4 with 5 
and 5 with 6), 7.2% of the students experienced no ch2mge in grade ^ 
equivalent scores, 33.3% experienced a gain of from a month to ai year, 
36.2% a gain of 1.1 or better, and 23.2% experienced a loss in grade 
equivalent. For the grade 5 with 6 site type B comparison, there wais a 
more significant decrease (.05 level) for the projected versus actual 
raw score differences than would be expected over a one-year period. - ^ 

Objective 7 was iltpvestigated to acquire information from field test 
facultires about skill mastery after instruction. Each of the 3j6 Levels 
A to G skills had been offered for instruction in the site type A schools; 
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across all strands and skills, the grand totat percent masters after 
instruction was 71.9%. Instruction for the 25 skills ehcoiw>a8sing 
Levels A to E had been of f ered. in the one site type « school from which 
data had been received, the grand total percent masters after instruction 
for this school was 60.<3%. * , 

Retention rates t^r skills mastered at break-in as well as after 
instruction were computed for objective 8. There was %n/ intermission of 
almost eight weeks (56 days) or more betweej^the time mastery had beep,,,, 
assessed either at break-ih or for postassessmeij* after instruction and 
the Center retesting fftt retention. ^ -4 

The grand total retention rate for skills mastered at break-in was- . 
90.9% for the site type ft sobtols and 81.5% for the site type ^*Bchools. 
The mean number of days between mastery and fetes ting for skills mastered* 
at break-in was 96.0 and 68.9 respectively for site types ft and B. The 
grand total retention rate for skills mastered after instrubtijwi was 
■ 73.0% -for hath site type groups, while the mean number of dayf^etw^en 
mastery and retesting for skills mastered after instructioiMps 87.1 for 
the site type ft schools -and 77.2 for the site type B echoes. The 
oyeraljisretention rate across both mastery at break-in ahd. mastery after 
instructipn^as d5.3% for the site type ft schools and 77.2%,for the site 
type B schools. ' „ 

. ■ . . > ' " ,. ■ ■ ' ■ ■ 



(3ie«fter.r Jl. p.v Asl^v, E., Hudson, B., ^ Ottb, W. Wisconsin Deaign 

• ;l National Con^uter Systems, 1974. 

'} CoBsptehm^iwWK^ Basic Skills , Monterey: CTB/McGiraW-Hill , 1968. 

}^6ap^^&^^e ' pirlaiary Tes ts . Jgyinceton: Education Testing Service, 19(65. 

r -f^t^ Askov , E . The Wisconsin Design for Reading Skill Development ; 
B^^iqnale and guidelines . Minneapolis: National Computer Systems < 
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STATEMENT OF SKILLS Wto' OBJECT • ^ 

FOR THE WISCCMSIN DESIGN. FdR READING SKILL DEV^lfe^MENT 
COMPREHENSION 1973-74 SMALL-SCALE F3SLD TEST 

■ *■ 



4f ^ 
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Statteent of Skills and Objectives for Coaprehension* 
Small*Scale Field Test Edition 



LEVEL A 

®** 
__ . ^ „ ^ . 

# 

Objective: The^'^i Id identifies the topic of a plltture. 

2. Determines sequence: first or last event>C/ ' ■ ^ k 

Objective: After he2u:ing a stozy, the child selects the one of 
four pict\2res that depicts the first or last event in the story. 

3. Uses logical reasoning 

- ^ 

Objective: The child selects the one picture Uiat djlu^cts the 
two-event activity described in an oral sentence. 'j 

b. Predicts outcomes 

Objective: The child sel^lbs the one pic ttire that <i^i<?t£r^^th^ 
outcome of an activity deapribed in an oral ^t^ncay \ 

. . -. - - - v^- - ■ v^J^ . - / 




UEVEL B 

1. Identifies a topic: paragraphs 



© 




Objective: The child identifies, the topic of a writt^^jttr 
selection. " .^^^ 

2. Determines sequence: event before or after 

Objective: After heeiring a story, th^^iild selec 
three pictures that depicts the event jBcurring inme 
, before or after another pictured event in the story. la i^'^ 

3. Uses logical reasoning: predicts butporoes^^-^ - v' > 

Obje'tftiye: llie child selects the one picture that depicl!k;^the^ 
outcome^^of a three-*event activity described in an oral g^fm:!^nce. 

* Objectives apply to the child's coqprehension of i^ltten aatc^|^l ^ I 
Levels C*G. Related prerequisite Objectives at Leyels A and B , are 'based; 
on pictures, orijjj^y presented materials, and written materials ^dalff arer;^ 
read aloud to the child (written-oral) . ^.^^^ ; > V 

Nunber in circle refers to skill' nuntoetj An 1 fpllowing'^he skill - 
number indicates that assessme'nt must be Tmlvidually au3minist6red. \l ' 
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4. Reads for detail® ' • . • v 4'' > . 

. ■ ■ • ■■■■■■ ■ ■■ . "^v . 

. yq^^f&f^ti^fei ■ The child answers questidns aboutvdetail fotihd iii a/ ., ■ 
' . ; writ|6h-6ral selection containing sentences written in the active ■ 

LEVEL C " '•■•^ ''^^ ■ i 

Identifies a "tppibt paragraph J<t) : ^ ^ 

Objective: The child iilentifies the topic of a writt^ ifelection. ' 

2. Determines sequence: event before or after^D 

Objective: After hearing a story, the child selects the one of * 
three statements that describes the event occurring immediately 
before or after another event described in a sta^eht taken 
from the story. ' 

3. Uses logical reasoning: determines cause-effect relationships® 

Objective: The child determines '^whether the relationship 
between two statements is cause-effect. 

4. Reads for detail . ^ 

a. Notes detail in sentences in active voice 



Objective: The chj^ answers questions aboiit. detail found in a 
paragraph con tainingT sentences written in tjtm^ctive voice. 



b. Ir^ferprets negative sentences^^ ^^Sf^^' 

Objective: The child determines whether a restatement ofV ' 
positive or negative sentence within a paragraph is true or - 
false. . 




The child Identifies, the topic sentence of H 



2. D6term^s sequ'ehce: explicit relationships^ 

• ; ^O^jective: Given one 6r two sentences containing two or three 
events with explicit sequential relationships, the child deter- 
mines ttie^ order of events. ' ' 

' 4^ " * ■ • • . 

^ # NOTE- Explicit Sequential relationships are identified by 

spf 9ific cue wor«as {&.q. , next, fixially) . 



8% 



if 



3. Reasons ^ductively w * A '^^^^ 



Objective: Given a major and minor premise, the child determines 
whether a given conclusion SLs correct. ^^^^^^-^ 



4. Reads for' detail 

, ' * (4) 

a. Interprets sentences with clauses at the beginning or end v-/ 

Objective: The child determines whether a restatement of a 
paragraph-embedded seijtence that has an introductory or ' 
terminal subordinate Clause is true or false. 

b. Interprets sentences written in .the passive voice 

Objective: The child determines \diether a restatenent of 
a sentence wri^|en in the passi\^ or active voice withiti a 
peiragraph is true or f ^Ise . ' ' ' ^ ' 

■ ■ . \ 

5. Uses context clues: unknown words n-^ 

Objective: ^e child uses the contextual device^ of direct 
d^^c^iption <imd cause-effect to determine the meaning of an 
in^a sentence. 



LEVEL g,: , ■ 

1. Identifies a toain idea: » paragraph 

Objective: The child identifies tiie m^iin idea of a paragraph 
with lib topic sentence. 

2. Determines sequence: multiple explicit relatiortshxpsV^^ 

Objective: Given thr^% to six senognces containing four to 
- 'six events stated witti an explicit sequential relationshifi, 
^* the child determines ithe order of events. 

3. Reasons deductivejj^^V^ '^^^ > 

* " v % * '■ ' ' - ' 

Objective: >G|iUeni'*two or three premises, either in <>r oti^ of 
otder, thei^ild determines wj^ether a given concliw 
correct^ • V 4 ./ • . > # ■ ■ ■ ' 

4. Reads for detail: sentences with one; pentrally embedded part /T\ 

* ^ (phrase or claxise ^between subject a(||i verb) v-/ 

Objective:^|g|phe child determines whether a restatonent of a 
paragraph^^edded sentence that has one centrally embedded 
ftort is true or false. 



5. .; .Use^ context clues: lanknown words 



Objective: The child uses the contextual devices of direct 
description and contrast to determine the meaning of an un- 
known word in a sentence. 

6. Determines the meaning of prefixes^ 

Objective: The child determines the meaning of the prefixes 
anti^, dis', for&-, in-, mid-^, mis-, mono-, non-, post-, pre-, 
re-, SGoii-, sub-, super-, and uni-, when used with a known word 
in coritext. 

^ ♦ 



LEVEL Fv, 

: 



1, ^ Identifies a main idea: two para^graphs 



® 



( 



Objective: T^ child identifies the main idea of a passage 
containing two paragraphs/ either or both of which may have 
topic sentencfik. * 

2. Determines sequence: implicit relationships^^ 

0\>jective: The child determine?^ the order of events from 
scra^|j|jLed statements which tqjgfelbier make a story with implicit 
sequential -relationshit)s. ^ W . ' . < ' 



NOate: Implicit, sequential reiatioriships are identified from 
the meaning conveyed in the passa?fe. 



o 



**Y.dl3g«s logical reasoning 



V|^a^ons deductively-i thtee premises®, '"^i^^ 



Objectj^ve: Given two or tAreev premises, either in or out of 
corder/ the child determine^ whetJier a given "conclusion is 
correct/ iAcorrect/ or indeterminable. ^ 

b* Recognizes an instance of a principle .l^^^'I 

Objective: Given a principle^ the fchild detennines whether 
specific events are instances of that principle. 

Roads for detail: sentences with one centraJ^Jy embeddgd pjart and 
^ * an introductory or termin^;j|jlause^ 

^ . ' ■ ■ ' '"''f. 

Objeoti^^ Jhe chiia determines wheiiier a, restatement of a 
V P*"9rapKi^M»edded sentence that ha^one c6ntrall% embedded 
part and an- introductory or terminaf^clause is true or false 
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5, Determines the meaning of siif fixes 

i> 

Objective: The child determines the meaning of thb suffixes 
-able, -an, -ee, -eer, — en, -<|ps, -fy, ^hood^ -ist, '^itia, 
"less, "let, -or, emd -ward, when used with a known word in 
cpntext. 

LEVEL G 

1. Orders events along a time line^^ 

Objective: Given a paragraph with -iiap licit sequential 
relationships, the child places the. events in the correct 
s*<^uence on a time line. 

2. Uses logical reasoning -0 

(2) 

a. Reasons dedu^tiyely V-/ 

Objective: llie child identifies and solves logical 
argtanents in context. ' ;^ 



b. Reaison^i^ inductively ^ 



© 



Objective i The child^ inf^ri^a general pr^^d£>le from a 
selectil^, in which- tfpecific^laices of inflRntion 
supportiil^ that general prin(|||ple are given,. 

... • ■ r ■ (4) 

3. %Uses context clues: obscure me2uiings.. of familisu: mrdsvZy 

-^Objective: The child determines the obscure meaning of a 
familiar word in contexj: by using the devices -of cause and 
effect, direct; description, and contrast.^ > 4 ' 

4. Determ^^es the meanii^g of pre^ajces>*>'^ ^ 

^ Objective: The child deterttllttes the meaning of the prefixes 

ante-, counter-- / de- ^en- , inter-, iiit^fa-, mal-, and trans-, 
t " when used with a knowft word in context. _ ' 

^< ■ • ''^'' "^ '■ ':■ 



. APPENDIX B * 

SCHOOLS PARTICIPATING IN THE WISCONSIN 
DES'iGN FOR READING SKILL DEVE^iOPNENT : 
COMPREHENSION 1973-74 SMALL-SCALE FIELD TEST 
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SCHOOL NAMB 


LOCATION 


SETTING 


ORGANIZATI(»l 


DESIGN 
ELEMENTS* 


Columbtis Elementcury 


Apipleton, WI 


small city 


multiage units 


WA** 


McKinley Elementary 


Apple ton, WI 


small city 


multiage units 

if 


WA, SS 

* 


Locii^t Lane Elem. 


Eau Ciaire, WI 


small city 


multiage units 


WA, SS 


•Whitney Elementary 


Green Bay, WI 


small city 


multiage miPts 


WA 


Victor Haen Elem. 

/ 


Kaukauna, WJ . > 


^^mall city 


within grade units 

# 


WA 


McFarland Elem. £Uid 


McFarlazid, WI 


suburbam or 


multiage units 


WA, SS 



Conrad Elvehjem 
Middle School 

Green Bay Avenue 
Elementctry 

Philipp Elementary 

Oregon *^ ^' 
Elementary 
Middle School 

Fairmount Elem. 



fringe 
Milwaukee, WI urbain 
Milwaukee, WI urban 



Oregon, WI 

4'' 
Golden, CO 



subtirbl^^ or- 
fringe " 



suburbam 



sel^i^^ai^fhed- 
classrocms 

se 1 f -cohtained 
claa^ocms 

■f- 

"$*'^ithin gxittde units 
multiage units 

self-contained 
■ classrocms 



WA ^ 

■ I 

WA, SS 

WA, SS 

WA, SS 



* In addition to Comprehensf&i at the^time of the field test 
** WA « Word Attack; SS = Study Skills ' ' 



# 
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APPENDIX C 



FIELD TEST MEMORANj^UM 
OF AGREEMENT ' 
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MEMORANDUM OF AGREi^NBNT 
/ betwet6n • ... 

The, Wisconsin Research^^-and peveJ,opm^t Cert ter for -Cogiiitivef^ Learning 



;;-v and 



|Th6 Wiaqonsin Research auid D^velo^Snent Cenl^ltv^^f^^ Cognitive Learning (Center) 

an4 .S (District) iigree cooperatively to field 

test during the 1973-74 acadeiidc year instructional materials for the Compre- 
hension element of ^e Wisconsin Design for Reading Skill Developments 

The field test will be conducted in ^ / 

— 'jj^ ~ ~ ■ 

^ School(s) with all children in their 

' ' ^ — f 

first through seventh (Gr. K-6) years of school. The Center agrees to supply - , 
all materials required by the SchooKs) . 

' • ; "■[:! ■■■ .... 

A. The Center will provide at no cbst to the District: 

1, All field tesV materials for teachers and students.^ • , 

2. Test scoring service for the break-lh te§||^ng of all participating >^ 
pupils. ^ 

3. Tests associated with the" .gathering of criterion data, 

4, An inservice program for two representatives -from each schciol. The 
» program will 1) acquaint tJiem with the field 

otHer instructional materials, 2) establish fiel^ teat purposes, 
3) provide information regarding implementat ig ii^iS ^^^ Q^the Comprehension 
eifeli*Shi ind ^) provide assistance relative td?::iMtruct^6^ methods 
and techniques. willj||||^olve one f\ill day^in Fallfflk73, and 
will be conducted at the mutual convenience of tto School emd the 
Center, Provisions for the inservice time will ^ shared by the 
DistricH eufid the Center, so thatej,coi^niently available school time ^^^^ 
can utilized v^erever possimiS and wherever it is not possible ^^^^^f^- 
the Center will relmbi^se the Representatives for itheir time. "^^-^^^^ 
rate of reimbursement will be at the hourly rate fox: pro:^essiona^x:J^;^ 



sta|f establishe* in the District salaury schedule. 



*Such materials ar/ provided for ; the exclusive use of the participating 
school atid ma/ i^t be reproduced for any purpoaf . other than field- testing 
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5, Consultant services as required. However, thd schodl^^system '-^ 
should recognize that ±he piirpose of the field test is to 
ascertain whether the product ceui be vised in & system wjrth the 
support only of the local centrail staff. If Center consultant 
services are required to any graat extent, the field test could 
be considered unsuccessful. ThSs is not meant to imply that 

we do not wish to know of gaps in the existing materials i we 
simply wish to indicate that the system is expected to supply 
the resoiucces at hand normally provided in support of . any- 
reading program. 

6. . Feedback to ^school syi^tems ^garding the field test results in 

the form of a written report and oral reports. Oral reports will 
be provided periodically at the mutual convenience of the school 
and the Centew An oral report will be available by January, 1975, 
witjp|a written report to follow by Janviary, 197^. 



B. .The District will ensile that eacjgppartia^pating school: 



1. Implements the Comprehension element according to the guidelines 
established in the materials and reviewed in the Inservice pro- ^ 
gram. \ 

2. 'Provides all non^ Design instructional matei^ials necessary for 
effective implementaition of the Comprehensraxl element. 

3. Makes available at least orie^ half-day of staff inservice- for 
participating teachers. This inservice will be cpnducted by 
the local leaders, who will have par ticipcrt:ed ip the Center- 
led, inservice. New teachers will receive additional orientation 
to the Design in its ^entirety. 

4« Engages all eligible K-6 pv^ils and' staff in the ^urti^ipating 
school i^ the program, ' ' ' v 

5. Devotes two hours weekly in eachj school to the teekbl'^j^^' of 
^ Comprehension. It is understood t^t the time devot^lfl to 

-Comprehension is not necessarily additional Curriculum time. 

The Comprehension skills are projected to be ^tegrated into 
. existing curriculum areas. Ins^uction will be base<l on t^e 
\"-;\, ^*b6htimious progress of the child without respc^^ to grade or 
"level" designations.^ 

6. Provides up to 2 hours of pupil .time for the fathering of 
criterion data yearly, prpvides' teachers to administer such tests, 
apprises the Center of the local testing program, and shares with 
the Center any intelligence or achievement data from the parti-, 
cipating school gathered through the system's testing program. 
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?• Coordinates the school system's testing program with the Center's 
^t^s ting program. 

, ■ 8. Pays any shipping costs for sending ^ tests to the Center for 
v processing. ^ ' >;;^ 

l', . ■ ' . • '"-^■'^ * 

^ 9. Informs the Center ik advance of scAool boundary changes 

affecting over 10% of the enrollment of a given schooi^so that, 
termination of the test at the e^ffected gpade levels can be 
jointly considered.; 



The term of this agreementj:^ will be from the time it is fully executed 
until June , 30 ^ 1974 . However , itii^ Cent^ir reserves : the right to 
gather foMfavij^bugh data until! iieoember ^^1^ 




Willi)^^!^^ush, Deputy Director 
Wiscp^^^afeaurch and Development 
Centf^R^ Cognitive Leeirning 



Agree to: 



(Signed) 



Date 



(Title) 



(District) 



(De^te) 




MICROCOPY Rf.SOLUIION IISI CIIARI 
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AGENDA FOR COMPREHENSION 
FI£LD TEST^iNSERVICE - 



■ ( 



AGENDA. 



t . ' ■ COMPREHENSION PIEM) TEST INSERVICE 

September 20/ 1973 
R&D Center, 1025^West. Johnson Street 
^ . ' Conference Area I - Rocms 247-253 

'- ' ' ' . ■ [ ' . ' ' ' ■ ' ' ■ 

Registration and Coffee / \^ 

Introductions 

Greetings' 

Skills: Rationale for Selection 

\ , . 

Comprehen&ion Strands - Sessions I and II 
Session I -. 

Session^frl \ V. ^ 

(Sessions I and 11 wi\l run concurrently. 
Those in Session I at !^0:00 will go to ^ 
Session II at 11:00 and vice^ versa.) 

• ' ' : ' 
Coffee Break _ ■ ^ 

^^COTiprehension Strands - Sessions I and l£. repeated 

Lioncheon - Union South, Roccn 211 . ^ 

Address: "Conmon Sense in the Teacl:jing 
of Reading" \. 

Field Test Expectation^ 

.% 

Assessment \ ^ 

. ■'}. . 

Local Insejrvice Plans 

Scheduling for Instruction 
Gr6Ui> A 
Group B ' 
Group C ' ^ ' 
Group D 

(Groups A-D will run concurrently,) 



Study Skills Report 
Inservice Evaluatioo^^ 




8:30-9:00 
9:00-9:10 
9:10-9:30 
9:30-10: Op 
10:00-10:45 

10:45-11:00 
11:00-11:45 
12:00-1:15 

1:30-2:00 ' 
2:00-2:30 

2 : 30-2 : 45 : 

2:45-3:15 ; 



3:15-3:45 
3:4^-4:00 
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^ APPENDIX E- 



SECOND VISIT INTERVIEW GUIDE: 
PRINCIPAL OR READING SPECIALIST 
AND UNIT LEADER OR TEACHER 
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B2 



Interview Gxaide / 
> / • ' 

Comprehension Small-Sea]^ Field Test* 

- / Second Visft/ 
, Principal or Reading Specialist 



School 



Date . 



Interviewee * 



Monitor 



Position 



INSERVICE 



1. Did you feel/ that the R & D Center inservice provided suf f ici^t < 
information ;!and materials about the Comprehension program to 
conduct ^our lepal inservice? ' 



2. If not, hoyf could the Centet inservice have oeen more helpful? 
(Be specific.) 



3* Were thez|e weaiknesses or problems associated with yoior local 

inservic^? If yes, please specify the ir^, nature; for example, was 



i|: long enough, complete enough? ^ 



4; ^Have thej R&D Center requisites for effective imp lessen tation, 

field te^st objectives, aifki schedule of field test activities been 
communicated to your sta^? 



1^ 



5. How mucn time ^s been or will be provided ^or keying local materials 
to the {teacher's resource file? ' 



) 

^ Inservice 
Other 



V. 



6* Has any specific plsm for keyifig materials b^en develojped? 



EKLC 
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' / • 

/ ' 

Principal/Readijig Spe<^i;alist 
Page 2 
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7. Who is, or will be, invoj-ved 'in the keying? 



8. Is the Compendium Sampler being utilized by your staff? 



SCHOOL READING PROGRAM 

\ 9 



10. 



What is the overall, official time allotment fc^r reading in your 
school (nvinutes per^day or minutes per Week),?* 



Grade If 
Grade 2 



Grado 3 



Grade 5 



Grade 4 



Grade 6 



Is language arts instruction included in the t^e allotted to 
reading, or does it have a time block of its own? 



Comment 



11. 



12. 



What instructional areas of the Wisconsin Design ar e not^ included 
in the reading or language arts time allotment? example , is 

time for Word Attack, .Comprehension, and Study Skills added on? 



Comment: 



Please describe the non- Design aspe^ts^DfyVour fi^ades 1-6 reading * 
program. Check all that apply and circle the gr&de.levelfs) to 
which they apply* v ' ' , 




School Reading Program 



Multitext * 
Publishers 



Graeme Levels 



a. 1 2 3 4 5 6 



One basal text (with supplementcury texts) b. 1 2 3 4- 5 6~ 
Publisher 



Basal - linguistic orientation 
Publisher 



Individualized (text or npntext) 
Language experience 



Other 



c. 1 2 3 4 5 6 

d; 1 2 3 4 5 6 

e. 1 2 3 4 5 6 

f . 1 2 3 4 ~5 



Ccxnment: 



< \ 



Priilclpal/Reading Specialist 

Page ^3 - < . ' - '■ \ ■ 

GENERAL • . ' ' • 

*13. ^ Do you have enough materials to implement the Comprehension program? 

How much (dollars) has been spent? ^ 



14.. Whajt, have you done about teachers new' to the Design this year? 



How many teachers are new to thJ- staff ar^d at what^'-grade levels? 



How many came from schools already using the Design? 



15, What do you see as the major obstacles tp ge^tingf the field test 
^ off the ground? ^ , . ^ \ 




■/■ ■ 

■ ■ a- 
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^Schopl 



Interviewee^ 
Position 



Enterview Gxxi^e . 
\ * ^ * * . - ^ 

Comprehensidij Small-Sc^i^ field l^e^^ 
Second Visi^ 
Unit Leader or TeacbeJ^ 
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INSERVICE ^ ■ I . 

1, Did your 16fcal Comprehension inservice P^^^^^® You wi^x sufficient 
, information and prepara^tion time Qo->jjiat^ skill gj-^^p instruction ^ 
could have 'begun soon aftervard?"* 



2. )lf not, how could the inservice h^ve bee^ helP you? 



r 



.MATERIALS ' . < ^ 

3, check all of the following statem^j^^ th^*^ 
a 



J.' 



true: 



y Time for keying locarl mat^^i^is ^^vided during 
local inservice. 



our 



c. 



The Ccmpendiiim Sampler w^q an a^-d ^Qying* 

The R&D Center had alre^^jy yaeV^^ ^^^^Is or texts that I use, 

I key materials as I'm teaching' ^^"'^.^^paring to teach, 
a skill. • : . I < 

I don't think that accurate key^^^ . ^ be done lantil I've 
taught and beciMne familiar y^ith ^ ^^^ll. * 



Statement £ is especially 
strands. ' , 



reas^n^'^g and detail 



keying is more easily and accur^*^^^^ do^e if ^ first Icjok 
over the teacher-directed acti^^^^^^ ^^^i^ a sk^H- 



Page 2 



Unit Leader/Teacher 

r 



h. The Teacher's Planning Guide provides sufficient how-to-^ 
do-it information for implementation , of the CcmJ)rehension 
program. (If you did not check "biis item, please use a 
Comment Card and be specific about the TPG's deficiencies,) 

_i. The Addenda provides an adequateodescription of the strands. 



Comments: 



IMPLEMENTATION ^ \ 

4. How many different skill groups eire you usually teaching at the ^gkne 
< 'time? ^ . , ' / 

5. Who decides whiclT skills you teacfl next? ^ ■ 

6,. On what basis i>s the decision* made (e.gl, profile card infontotion) 



^ I iL. 
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It Jls'-^e decision "based on^oxir own homeroom' s skill needs, or on the 
basis^'of a larger group of children? 



What usually happens to. the lowest 10% pf the reapers? 
; \ ' a . special grdup. formed to accommodate themX 

b . work with higher group above their abilit:^ level 
c. often dpi^'t fit and do independent work 

d . have tutoting (one-on-dne) at the afppropriate level v 

*» . . * ■ < 

^ _e. work with* younger children who shar^ their common skills 

heeds fT 



9. What usually happens to theShi^^est 10% of^he readers? 

. ' ■ • i 

a. special group formed to accanodate them ^ 



^ b. work with lower group below their ability level 



c. often donVt fit and do* independent work 
,d. act as tutors - ' 



e.'work with older children %ho share their common skill needs 

103 ' 
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Unit" Leader/Teacher . " ' - ^ , ^ 

Page 3 •''*.. 



10. Did you select one of the scheduling for rias true tion plans offered 









V 














\ ■ 












If not, wha^ schedule did yqu use? 









11. What skill(s) dM yo^^ teach during the , last skill grcmp 'Cycle? 



To what age children? 



\ 7^ 

12.^ How many days did your skjjil group last? 



How many minutes did ,it meels-each day? 




13. Check all of the following s^tements that are true: 

a. My skill group was formed by skewering the profile cards. 



\ 



Only nonmasters received instruction. 



c. used the straut^^oncept and taught the same skili at 
more than one level. ^ k ' 

Comment: " ^ ^ 

The teacher-directed activities in the teacher's resource 
file sufficiently supported instruction. 

' / i 
I had to supplement the TRF. - ^ 



. , My students enjoyed doing^ the activities from the TRF. 

jg. The activities in thei fRF were top diyfficult or frustrating 
for some of my students-. (Be specific with a Comment Card.) 

Ji. Tl^e skill group should have lasted longer (or been shorter) . 

I postassesssed after instruction. 



* 



j. l^was pleased with the number of fetudents mastering the 
skill af ter ^instruction. 

_k. I* have confidence in thei assessment program. 



Conments: 
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Unit Leader /Teacher ^ 
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14. do you usg, or do you think you will ^ 

15. Are the profile cards kept up-to-date? 
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PRINCIPAL OR READING SPECIALIST, 
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Interview Gul^e 



Comprehension Small-Scale Field Test 
Third Visit 
Principal or Reading Specialist- 




Position 

1. Are all classf ocm teachers (Greuies 1-6) in your bulldingNteachifig the Design' 
. Conqprehenslon skills? 

. . , — rr - ' 

Do kindergeurten children receive Instruction in word attack skills? 

Did K teachers start Level A 



Comprehension this semester? 



3, Does your school have instructional aides wh^are involved in teachin 



r 



the CompV^ension skills? 
what grade, levels? 



7^ 



Which of the following responsibilities are they usually assigned with, 
regard to Comprehension skill instruction? ^ 

• supervise students involved in independent activities 



assist If teachers Insljiill Instruction 
teach skin groups • 



jprepare materials' f Q^keachers to use' 



key materials for teachers to use 



_fotm skill grqups (skewer , c^i^rpte, etc.) 



atlininister tests 



score tests 



notch -prof lie cards 



1Q7 



Other? \ 



Principal/Reading Specialist 
Page 2 ^ 



4. ,For the field test; report we need to know when (month) Comprehension 

" skill ins tr\i:tion began for: ' . 

Grade Month 

2 . 

3 . . [ 

4 • ■ ^ 

' - ' -5 • 

6 • 



5. Have any teachers changed either their implementation plan or schedule 
since they first started? . ] 



r If yes, please specify the changes, wjjen (month) they ocbiirred, at 
•v^ which grade level (s), and the number of "teachers in^^lved. 



6. How would you rate teacher reaction to the Design's comprehension program? 

REACTIC»IS 

STAFF Negative Lukewarm Positive Enthusiastic 

Most . 

Some . ' ' 

j> " . 

A few _____ ' 

Comments? 
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Principal/Reading Specialist 
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7. What specific recommendations would you give to the Center for revising 
the Qaniprehension program? ^ 




■ ) 



ERIC 



Principal/Reading Specialist 
Page . 4 ^ 



Questions 8-13 should be answered only^; by tho^e schools that are iin£>le- 
menting Study Skills in addition to Word Attack and Comprehension. 

— ■ ^ 4 ■ — ^ — 

We are interested in learning abotit overall Design implementation in schools 
.that are now impleraenting three of the elements — Word Attack, Study SkllJ.s, 
emd Comprehension. Some schools, for exeunple, have a "skill3 period" every 
day during which staff members alternate cycles of Word Attack emd Compre- 
hension, or cycles of Comprehension emd Study Skills, etc. Some staff |pnem- 
bers do not form skill gro.Cijis for Study Skills, although they provide 
instruction in the Study Skills. Qi;estions 8-13 deal with management of 
the Design when staff member^- cure implementing more than one element. ^ 

8. On the chart beloW plea3e check, by grade level/^ all Design elements 
for which there are formal skall groups (studentfs are grouped on the 
basis of non-mastery, receive instruction, and are postassessed. ) 

DESIGN ELEMENTS 

GRADE Study Skills 



LEVEL Word Attack Ccmpjifehension Maps Graphs-Tables Referenbes. 

K . ' ^ \ ■ 

1 ^- ^ \ ' _^ 

2 ' 



We would like to know more cUtx>ut th^ lines that you left blamk on the 
chart in question 8^. For ^ example, did you leave a line blank because 
no instruction is being offered in that element? Or did you leave it . 
blank because teachers are offering skill instruction in that element^ 
but not in formal skill groups? 

Please go back to the chart and enter a "1" or a "2'| on the lines that 
you left blank. 

1 = no instruction is being offered 

2 = instruction is offered, but not in formal skill groups 



110 



^4 

Principal/Reading Specialist- 

Page 5 f ' ' "" ' ' x!l' 

* ' * -, " 

10. , Do any grade levels or units alternate cycles o£ skill instruction in 
one Design element with another? ' . . ' , ■ 



For example: 

i'lternate' cycles of Study^ Skills, Word Attack^ and COTiprehension- 
instruction? . ^ - 

If "yes", "at .what grade levels? 



Alternate Ccniprehension and Study .SkUis? 



If "yes", at whatf grade levels? 



Alternate Comprehension and Word Attack? 
If "yes", at what grade levels?' _^ - 



Alternate Word Attack and Study Skills?* 
If 7yes", at what grzide levels? - 



11, *Are\any grade levels or units grouping so that a student receives skill 
instruction every day in Word Attack, Comprehension, and Study, Skills?> 




Your .school has be6n involved in the Design field test since 1970, Staff 
members at your school are implementing more of the Design.' s elements (that 
is. Word Attack, Study Skills, and Ccsnprehension) ^ than most other schools. 
They have also been- implementing for a longer "period of- time than most other 
schools. In view of yoiir experience and understanding 'of the program now/ 
please emswer questions 12 and 13 which deal with projections' aibout overall . 
Design implementation. 

12. Is there enough time in a school day to schedule formal skill instruction 
in Word Attack, Study Skills, and Comprehension without xising alternate 
cycles? '. ' , 



13. If you answered "yes" to question 12, is it iTeasonable to ask teachers 
to form skill groups so that'^a student is in a Word Attack group, a 
Study Skills group, and a Comprehension group regularly (without 
alternating cycles) ? • 

If "no", what is reasonable to ask teachers to do? 



Ill 



Interview Guide 



Comprehension Small-Scale Field Test i 
* Thijfd Visit ^ . 

Unit Leader or Teacher / \ 



School ^_ Date 



Interviewee ^ \ Monitor 



Position, 



INTERVIEWEIE : Please jn£u:k yoiar amswera ahead of time to question 20 (printed 
on green). You may wish to familietrize yoiu3seif; with questions 
1-19 which will be asked during the interview. However, it is 
' ; not necesseury for you to write in answers to questions 1-19. 



1. From which of the ^following groups 2ui% your, Ccanprehension skill groups 
fomed? ' ^ , ' ' 



the whole grade level 
the whole unit 
my homeroom 

.my developmentar reading groups ""P^ 




Other? \ 

2* 'Who forms the skill groups? 

unit staf^ ^ 

^ lanit leader 

ihstructional aide 

homeroom teacher 

^ developmental reading teacher 

Other? < , 



roil 



3» Are skill groups formed on the basis of information from profile cards 
or charts? ' 
are they formed? 



or charts? ". _^ ^ If not, on what basis 
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A 



Unit Leader/ Teacher 
Page 2 



/ .■ 



^ . • • / , 

4. Wiat usually haJppens to a student v^en he isyklready a master of the 
skill (s) you or your unit are teaching? /, / L 

, ■ he does independent work in . / ^ > ^ - ; \ . 

^ he is^ ground for instruction on^/a skill' .he has mastered 

. • ^ • / • ■ ■ ■ ' . , 

he is grouped for inistructioA/on* a' skill ^e ha& not mastered; 

with children Sifrom another vaait or grade levejT . . 



Other? 



/ 



5. How many Comprehension skiM/;groups_cu:e- you usually -teaching? 



one group 
two groups 



other 



three qrovcp^ 



7 



to. each grouu^ / / 



How ^any skills* are you usual]^ teaching- 



6.. How Long dp yoUr Comprehension skiJ^ groups usually last? 



days for 



/ 



minutes a d 



How ofycen do* you tasually regroup for Qomprehension skill instruction? 




/ every two w^eks 
every three weeks 
every ^f bur weeks 
every six weeks 



ther? 



\ 



8./ Do yoti alternate days or weeks of . ins^triiction for: 

" - * .. 
Word -Attack and Ccmiprehension? ' t - 

Comprehension auid basal? - 



Coqiprehension cind something else? 

/ • , 

Other? ■ ■ ■ : , 
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Unit L«ader/Toacher | • . / 



9. For any skill, do all the students in your group(s*) receive the same 
instruction, or do you sixbgroup> within a group, using different 
materials with the subgroups? ' . ' ' 

10. Do masters as well ai^ r^orimasj^rs sometimes- recei^^e ins^juction in a skill? 



At what times? 



11. Have you used the strand concept in skill^nstruction—giving instruc- 
tion in a skill at more than one |skill level? ^ 

12. In teaching the Design' s'^ comprehension^ skills, how much do you lase^ each 
• of the materials listed below? ^ 

/ / J. . . , NONE LITTLE SOME MUCH 

teacher-made materials \. 

(games, worksheets', etc.) - 



act;ivities from 'the Comprehension 
Teacher's Resource Files (TRF) j 

b^al workbook pages you like 0 

commercial materials that were keyed 

for the TRF by the 6 & D Center ' 



commercial materials that were .key^d ^ 
for the TRF by your school staff --^ 



Other 



13. Have yo^^ added materials to folders in the teacher's resoutce Mies? 



14,. Do you test for mastery after instruction? 



ruction? - 

^ -Which test form do you use? . ^' ■ Form 1 ' Form II 

15, Are the Comprehension profile* cdrds- or chairts kept up-to-date? 
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Unit^ Leader /Teacher 
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16. Do new students receive br€adc-in testing fSfc\ Comprehension? 
How Boork aure they usually tested after they '^enrolled? 



17. Have any students finished a skill level since bre£ik-ln testing? 

Di^you use pretesting stg:*ategies with these students to see what skills 

they may have acquired at the next level? 

If y^f hov did the pretesting sj;rategies work? ' 



|L8. Who 

progress? 



D takes responsibility for followi^ 



L 



vidual student's ski 




19. What specific reccmmehdationsyto$^dSyou give to the Center for revising 



the Comprehension prbg|fim? 



ERIC 
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Unit Leader/Teacher 
Page 5 



20. Please rate how you feel about statem^J^g a"?* 



a^^ The Wisconsin Design hai identified 
skills that students need for ^ 
heading ' comprehension. 

b. Students do not need instruction^^ 
comprehension ^skills they've 
already mastered. 



Disagree Hr£22S^ g gutral Agree^ 



Strongly 
Agree 



d. 



Mastery of* the objective^ .in the / 
Main Idea strand i& essential for 
reading comprehension. 

To effectively teach canclpreh^nsio^ 
skills, a .basal reader program is 
needed a^long with the Wiscon^ 
Design . 




Mastery of the objectives- ifl thfe 
Affixes strand is essential J^r 
reading comprehension, 



Master^ of the objectives;in the 
Rfeasoning strand is essential for 
Reading comprehension. 



■4. u^^ing informal assessment (teacher 
judgment or observation^ is as ef" 
fective -as formal assessment (tests) 
for judging skill mastery. 

h. Spending extra time to teach the 
Design's comprehension . skills is 
worthwhile. ^ ^ 

i^ T)ie tests (Forms I and II) measure 
- the comprehenfeipn 'behavioral 
objectives. 

j. The Wisconsin Design alone is an 
effective program for teaching 
comprehension skills. 
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Unit Leader /Teacher 
Page 6 



r 



I 



strongly ' Strongly 

Disagree Disagree Neutral Agree Agree . 



k. Mastery of the objectives in the 
Context strand is essential for 
reading ccmprehension. 

1 . The Wisconsin Design exiables me 
to, teach comprehension skills 
. bettet than any other program 
I • ve . vised . ' 

m. Mastery of the objectives^ i^ * 
the Sequence strand is essential 
for reading comprehension, 

n. students can apply the crompre- 
hension skills they've mastered, 

o. Mastery of the objectives in the 
^Detail strand is essTeritial for 
reading ccmprehension. ■ \ 

p. Teachers should help studentsk^ 
^ apply the ccmprehension skills , 
they have mastered. 
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AND UNIT LEADER OR TEACHER 
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101 



118 



102 



interview Guid^ 

1973-74 Conij^ehension Stnall-SC^^® ^ield Test 
'^-^ Fourth Visit 
Principal or Reading Sp^^^^^ist 



School V 



Interviewee 
Position 



Date 

Interviewer 



IS. Do you have any new^taff members this year? 

no ' ^ 



yes , , 

a.' At what grade levari? 



b. What kind of inservice did they ^^^^ive? 



2S. Has there been any staff reorganization this year? For example, 

(a) Teachers^ssigned • to a different 9rade level this year 

' ~r . r ' ■ 

(b) Units reorganized (multiunit school; 

(c) Different unit leaders (multiuni*^ school) 

(d) Other * - 



If any of a-d are true, explain the chang^^- a. 



JJo staff reorganization 



3M, During the last yeajr were commercial materials order^ specif ically" for 
use with the Design's Comprehension progr^m^ 



Yes 
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Principal/Reading Specialist ' • ' lj63 

Page 2- " . 

v ^ ' ' ■ ^- ■ 

5M. ptiring the suinmer, or sTnce school has started, was or has any sp^^^ 
time-bejpn set aside for kej^^J^^ <>r greparation of teacher-made materiaas 
fojf^^^Ae, Comprehension pro^^raro? * 

No^ specific time not set aside; keying, etc. is ongoing. 

' ^ , •••• ^ • . ' ' \ " ^ ' ■ - • 

yes, specific time set as (Explain: Funded summer project?' 

Summer inseryice time? Fall inservice tim6?) 



61. Has Design Comprehension skil\instructiori already started at all grade 
levels or in all units (multiunit school^)? 

Yes . ^ ^ 

How long after school has started do teachers usually ^wait before* 
beginning Design skiJ-l ij^struction?' ' ...... .. 



No. Who' (grade levels or units) an<J when do^ they P^Lan to start? 



y ' . ' ' ^ 

71. Do.^you know whether any teachers ©r^ units^iiave altered last year's 
strategy regarding the impleinentation qf Comprehension? 

' ^ Don • t know . 

'/ All teacher§ or unit§ implementing the same as last year 



SOTie, teachers or' units imp lem^ting differently fron^ last year 

For whom differen t (what grade levels or units) and how different 
(changes in: skin gro^Ps-composition and number of days they 
meet; time block taught; time sperit each day, etc.).-^ 
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Principal/Reading Specialist 
Page 3' 



81. How did this ye^'s grade level 1 children "^reak-in" for the Design's 
Comprehension program? . . ' 

■ V. , . , ^ ^ • 
(a) Don?t kncM .. • ■ ; 

(b) They'*re not in the program yet 

^{c) They received break- in testing in September A 

.(d) They started the 'progrart in kindergarten: 



(1) administered booklets for break-in, received instruction 
postassessed 

(2) no pretest (break-ih) > all rjeceived instruction, all 
poptassessed" ^ ■ " 

(3) no' pretest (brfecUc-in) , all receive^instruction, no 



^ posttest . 

Other (include any variations iA above categories) 



ass 



r 
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Interview Guide 

1973-74 comprehension ^all-Scale Field Test 
Fourth Visit 
Unit Leader or Teacher 



School - . ' . Date, 



Interviewee > • ^ Interviewer . 

Positibn . . . « 

■ . ^Q g ^ ■ ^' 

NOTE: If interviewee teaches grade level 1 o:^^ is involved withr^rade level 1 
students, aak question 8 f rent 'Principal/Reading Specialist Xwter view 
. Guide. " * ' . 

^ •• ' ■ ■ , ,. ^ ■-. ■ 

c3^T.I. Can you briefly review how you were implementing the Dtesign's Comprehension 
program last year? (how of ten/regagpup; how many weeks did skill groi:upings 
last; how many minutes groups met each day; did you "alternate cycles;" 
groups formed within homeroom or vinit, across honerooms; taught in ' 
isolation of in reading group, etc.) 



erIc 



sl^Eoer 



21. Are you continuing with last ye£u:*s implementation plan, or have you 
altered it? 



(a) Continuing with last yeaur'^plai^ 



Have you started Comprehension skill instruction? 
Apj^roximately when (week of school) did you start? 

(b) Altered l^st year's implementation plan 



What are you doing differently? ■ ■ * 



What caused you to change? 



Have you started Comprehension skill instruction? 
Approximately when (week' of school) did you start? 

, 122 



Unit lieader /Teacher^' 

Page 2 ^ 



31. Have you noticed whether any students have experienced skill loss or 
^kill gain (Comprehension skills) over the svamner? 

Haven't really noticed or locked for it 



Looked. How did you check for it? 



What did you find? ' S kill loss , S kill gain 

For^any skills in peurticulau:? ; 



I If skill loqip, what did you do about it? 



4T. How have you placed this Fall's new student into the Comprehension 
program?^ 

(a) New students receiving skill instruction; no pretesting or 
' battery testing planned 

(b) Break- in. testing via pretesting strategies. How was level to 




administer d»^rmined?' 



(c) Break- in testing via battery. How was level to administer 
determined/^ 



Other: 



5. How do you feel about continuing with the Desi^'s Comprehension program 
^ (negative/ lukewarm, positive, enthusiastic)? 



Are you experiencing any peurticuleur prdblema? 
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Date 
Name _ 

School 



Comprehension 
Ccxmnent Caxd 



Material affected (be specific, e.g., include page number, level, skill 
nuniber) . 

gueation/Problem / 



Do you have any reconaoendations? 



Success 
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postassessmbMt following 

INSteUC3?10M POKM 



109 



128 



NfyfEjt This quostionnairo should be filled q^t throughout this school year 
b^' diich- LeacherVwho is participating in the Comprehension small-fcale field 
testis Return of this questionnaire to the R & D Center will be requested 
in l^ay, 1974. • 



COMPREHENSION SMALL-SCALE FIELD TEST 



School 



(fill in school nazne) 



Directions: Use a different line for each skill that you teach and postassess 
Enter -the information requested under each coliimn. heading. 



^ POSTASSESSMENT FOLLOWING INSTRUCTION 




Skill 
Level Niimber ' 



Date 
Assessed 



Nimiber In 
skill groiqe>' 



Result 
Masters N^nmasters 



pie: ^A 



10-26-73 



20 



127 



16 



Matlpnat Evaluation Commitfaa 

Fcancis S. Omm, Chaimn 
Emetilus Profevsoc 
Uniwtsity of Chicago 
HcUn.Bain 

PMrff Pircsidcfit 

))^tion«l Education Aatocuition 

Lyie Bourne 
PrafieOTOc 

Univcisity of Colocado 
\Su« Bucl 

CoMultMit. Portland. Ocegon 

Roald F. Ca>q>ben 

EaMiitus Piofessof 

The Ohio State Univctaity 

Gcotge E. Dkkaon 

Dean. Collegt of Education 

Univctaity of Toledo 



Lany R. ( 
Profeaabt^ 
Univetaity of lUtnoia 
Cheatet W. HanU 
Profeaaof » 

Univetaity of Califotnia - Santa Batbeta 
* Villian G. Kateenaieyer^ 
Piofeaaoc 

Duke Univetaity 

Baibata iThompaon ^ 

Supetintendent of Public Inatiuition * 

State of Wiaconain 

Joanna WilUama 
, Profeaaoc 

Teacheta College 
' Columbia Univetaity 



Uni¥«rsity Advisory Committaa 

John'R. Pabnet. Chaiiwan 
Dean 

School of Education 
William R. Buah 
Deputy Ditectof 
R&DCentct 

David E. Cionon 
Dean 

CoUcgr of Letteta and Science 

Diane H. Eich 

Spccialiat 

R&D Centet 

Evelyn L. Hoekengh 

Coofdinatot 

R&D Centet 

Dale D. johnaon 

Aaaociate Ptofeasoc 

Cuiticulum and Inatmction 

Hedwtt j. Kbuameiet 

Membet of the Aaaociated Faculty 

R&D Centet 



Jamea M. Upham 

Membet of the A«aociated Faculty 

R&D Centet 

{Irayne R. Otto 

Aaaociate Dicectoc 

R&D Centet 

Richard A. RoaeaiiUei 

Ditectof 

R&D Centet 

Elizabeth J. Simpaon 

Dean 

School of Family Resources 
and Cofisttntet Sciaocea 
Len Van Eaa 

Associate Vice Chancellot 
Univetaity of Wisconsin • Madison 



Associatod Faculty 

Vernon L. Allen 
Ptofessar 
Paychology 
B. Dean RowIimi 
Piofeamif 

Educational Administration 
Thoaua P, ('jitpentet 
Aasiatant Piofesaor 
Cutriculum and Inatmctidn 
Marvin J. Fnith 
Profeaaur 

Educational Administration 
Jnhn C. llarvey 
Ptofessoc 
Mathematics 

Curriculum and Instruct icxi 

Frank II. Ilmiper 

Profeaaui 

Child Development 

Herbert J.B tous meier 
V A C. llena^ Pripfcssur 
Educational Psychology 
Joseph T. Ljmton 
Aaaiatant Profeaacir 
Edutatinnal Psychology 



Joel R. Ijevin 
Professor 

Fducati<»inl Psycholiigy 

L. Joseph l.inH 
professor 

Institutional Studies 
James M. I.iphnm 
Professor r 
' Edu ca t juna l Admintstral ion- 
Donald N. Mclsaac ^ 
Professor 

Educational Administration 

Gerald Nadler 
Professor 

Industrial Fjiginccring 

Wayne R. Otio 
, Professor ^ 

. C'urriculum and Instruction 
Robert 0. Petzold 
Professor ' 
Music 

Curriculum and Instruction 



Thomas S. Popkewiti 
Assistant Profeaaot 
^Curriculuii! and Instruction 
Thomaa ^. Romberg 
Professor 

Cuniculum. and Inatruction 
Richard A. Rossmiller 
Professor 

Dennis W. Spudc 
Assistant Professor 
Educational Adminia teat ion 
Michael J. Subkoviak 
Assistant Professor 
Educational Paychology 
Richard L. Venezky 
Professor 
Computer Sciences 
J . Fred Weaver 
Professor 

Cuniculum and Instruction 
Larry M. Wilder 
Aasiatant Professor 
- Child Devetopmetif " 
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